(19) 



J 





(12) 



(43) Date of publication: 

27.01.1999 Bulletin 1999/04 

(21) Application number: 98113679.9 

(22) Date of filing: 22.07.1998 



EuropSRies Patentamt 
European Patent Office 
Office europton des brevets (11) E P 0 893 308 A2 

EUROPEAN PATENT APPLICATION 

(51) Int. a®: B60R 16/02 



(84) Designated Contracting States: 


(72) Inventors: 


AT BE CH CY DE DK ES R FR GB GR IE IT LI LU 


• Kubota, Tomoki 


MCNLPTSE 


Arakawa-ku, Tokyo-to (JP) 


Designated Extension States: 


* Hori, KbJI 


ALLTLVMKROSI 


Bunkyo-ku, Tokyo-to (JP) 




• Mazda, Manabu 


(30) Priority: 22.07.1997 JP 212491/97 


Ictilkawa-shi, Chlba-ken (JP) 


27.02.1998 JP 64414/98 


• Adachi, Kazuhide 


12.03.1998 JP 82626/98 


Nerima-ku, Tokyo-to (JP) 


13.03.1998 JP 82710/98 


• Mukal, KouJI 


23.03.1998 JP 95386/98 


Suglnami-ku, Tokyo-to (JP) 


23.03.1998 JP 95387/98 




(71) Applicant: 


(74) Representative: 


VOSSIUS & PARTNER 


KabushikI Kalsha Equos Research 


Siebertstrasse 4 


Tokyo 101 (JP) 


81675 Munchen (DE) 



CM 
< 

00 

o 

CO 
CO 

o> 

00 



(54) Device mounted In vetiide 

(57) A device is mounted in a vehicle which allows 
one or more of personified agent or imaginary living 
body to appear before a driver or passengers in the 
vehicle for communication therewith. A current status 
sensor (40) detects predetermined vehicle current sta- 
tus data, and a memory (292, 293) stores study data in 
response to entry of the latest cun'ent status data. One 
of a plurality of communication programs stored in a 
program table (291 ) may be selected In accordance with 
the current status data and/or the study data. /\n agent 
control unit (11) controls activity of the agent in accord- 
ance witii ttie selected communication program. The 
determined agent activity Is outputted tiirough a display 
(27) and a speaker (25) for communication with the 
driver. 
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Description 

This invention relates to a device mounted in a vehi- 
cle, and more particularty to a device for controlling 
activity of a personified agent on a display in a vehicle s 
room for communication with a user such as a driver. 

Various audio instruments such as radio, cassette 
tape player. CD player. MD player, etc. are installed in a 
vehicle to provide comfort and relaxation to the driver 
while driving the vehicle. This is only one-way communi- 10 
cation. 

Bi-directional communication between the driver 
and his or her friends outside of the vehicle may be 
achieved by using radio communication medium such 
as radio transceiver and portable telephone. However, is 
the driver have to find out a companion to talk to by 
operation of the radio communication medium, which 
might be dangerous during driving. The companion out- 
side the vehicle can not understand the vehicle and 
driving condition that the driver meets with. 20 

In Japanese patent laid-open publication Na 9- 
102098, specific infbrmation is transmitted to tiie driver 
through con'espondlng expression and/or activity of an 
agent appearing on a display. In accordance with this 
prior art. the same expression and/or activity of the 2s 
agent is appearing in the same situation. Thus, this pro- 
vides improved visual literacy but is still a kind of con- 
ventional meters. 

Accordingly, it is an object of the present invention 
to provide a novel device mounted in a vehicle that pro- 30 
vides bi-directional communication with the user in the 
vehicle. 

Anotiier object of tiie present invention is to provide 
bi-directional communication witii the user in a vehicle 
by means of a personified agent whose activities are 55 
determined not only by the current vehicle condition but 
also by tiie results of past record study. 

To achieve the above object, in accordance with the 
present invention, a dance is mounted in a vehicle 
which allows one or nxire of personified agent or imagi- 40 
nary living body to appear before a driver or passengers 
in the vehicle for communication therewitii. 

In accordance with an aspect of the present inven- 
tion, the device comprising first sensor means for 
detecting predetermined current status; first memory 45 
means for storing study data reriewable in response to 
entry of the latest current status detected by ttie first 
sensor means; at least one personified agent communi- 
catable witii user in the vehicle; second memory means 
for storing a plurality of communication programs; com- so 
munication program select means for determining one 
of the communication programs to be executed in 
accordance with the cunrent status detected by the first 
sensor means and/or the study data currentiy stored in 
the first memory means; agent control means for con- ss 
trolling activity of tiie agent in accordance witii the 
selected communication program; agent output means 
for outputting the agent's activity to the user in tiie vehi- 
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de detemiined by the agent control means. 

In a pretended embodiment, the first sensor means 
detects at least one vehicle condition selected from the 
group consisting of a current vehicle position, a current 
vehicle speed, a current shift position, winker operation, 
wiper operation, a coolant tenperature. an engine oil 
temperature, an engine oil pressure, a passenger room 
temperature, an open air temperature, ignition switch 
operation, a brake pedal stroke, an accelerator pedal 
stroke, side brake operation, horn operation, belt loos- 
ening, a fuel remainder, light or lamp operation, seat 
belt operation, door opening, window opening. ABS 
operation, air-conditioner operation, distance to tiie pre- 
ceding and succeeding vehicles and decrease of air 
pressure in tire. 

The user's own data may be included in the current 
status. The user's own data may also be considered in 
determining the communication program to be exe- 
cuted. 

The device of the present invention may further 
comprises second sensor means for detecting user's 
response to tiie agenf s activity outputted by the agent 
output means, and third memory means for storing 
user's response data detected by the second sensor 
means. In tiiis embodiment, the user's response data is 
also conskiered in determining the communication pro- 
gram to be executed. The user's response data is sub- 
jected to study so that tiie study data in tiie first memory 
means is also renewable in response to entry of tiie lat- 
est user's response data. 

The agent output means may comprise one or kx5tii 
of display means and speaker means. In a preferred 
embodiment, utterance pattern of the agent outputted 
by the speaker means is selectively determined by the 
agent control means. In another preferred embodiment, 
appearance pattern of the agent outputted by the dis- 
play means is selectively determined by the agent con- 
trol means. 

The device may include agent's character select 
means for determining a specific character to be given 
to tiie agent. In this embodiment the agent control 
means determines tiie agent's activity in consideration 
of tiie agent's character determined by tiie agent's char- 
acter select means. 

The device may include fourth memory means for 
storing data indicating user's familiarity to agent opera- 
tion of the device. The agent control means determines 
ttie agent's activity in consideration of the agent's famil- 
iarity stored in tiie fburtii memory means. 

In anotiier preferred embodiment the agent output 
means allows a plurality of different agents to be distin- 
guishably recognized one anotiier by tiie user in ttie 
vehicle. For example, tiie device includes fifth memory 
means for storing sut>-agent program, which is selected 
in correspondence to a specific communication pro- 
gram stored in tiie second memory means so tiiat a 
sub-agent of the selected sut}-agent program is output- 
ted by the agent output means together with the main 
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agent of the executed communication program. The 
sub-agent may be a hetp-agent explaining the activity of 
the main agent The sut>-agent may be a short-cut 
agent allowing short-cut of operation executed by the 
main ag»Tt communication program. Preferably, the 5 
sut>-agent program t)ecomes operative when data indi- 
cating user^ fanrtiGarity to agent operation of the device 
is greater than a predetermined critical value. 

In another preferred embodiment, a background of 
the display means is selectively determined so that the 10 
agent appears on the display means together with the 
selected background. 

The above and other objects of the present Inven- 
tion will become apparent from the following description 
when read in conjunction with the acconpanying draw- is 
ings in which: 

Fig. 1 is a block diagram showing an agent system 
in accordance with a first embodiment of the 
present invention; Fig. 2 is an example of a program 20 
select table; Rg. 3 is a table descnbing agent's 
activities to be outputted when a specific program 
number is selected in reference to Fig. 2; Figs. 4A- 
4D represent a series of an agent's activity shown 
on a display by way of example; Fig. 5 is an exam- 25 
pie of a study item data in an agent data memory of 
Fig. 1; Figs. 6A and 6B represent an example of 
response data In the agent data memory; Fig. 7 
shows various data files stored in a navigation data 
memory; Fig. 8 shows various sensors in a current 30 
status sensor; Fig. 9 is a flowchart of the agent con- 
trol operation executed in this emtxxiiment; Figs. 
1 0A-1 OC show an example of the agent activity out- 
putted in this embodiment; Figs. 11A-11C show 
another example of the agent activity outputted In 3s 
this embodiment; Rg. 12 shows contents of the 
agent data memory in a second embodiment; Fig. 
13 is an example of an agent character select table 
in the second emtxxiiment; Rg. 14 Is an example of 
a voice oulput pattern select table in the second 40 
emlxxjiment; Rg. 15 is a flowchart showing a main 
routine of the agent control operation in the second 
embodiment; Fig. 16 is a ftowchart showing a sub- 
routine executed at step S1 14 in Rg. 15. Rg. 17 
shows contents of tiie agent data memory in 45 
another modified embodiment; Fig. 18 is a flow- 
chart showing the agent control operation to be car- 
ried out at S1 14 in Fig. 15; Figs. 19A-19C show an . 
example of the agent activities outputted in this 
embodiment: Rg. 20A represents a example of a so 
program select table; Rg. 20B represents the 
agenf s activities to be outputted in the communica- 
tion progpram selected by ttie table of Rg. 20A; Rg. 
21 shows contents of the agent data menwry in still 
another nrKxlified embocfiment; Fig. 22 shows an ss 
example of the familiarity data In the agent data of 
Rg. 21; Rg. 23 shows contents of the agent data 
memory in still another nwdified emtxxiiment; Rg. 
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24 is an example of the help-agent select table in 
the agent data menrxsry of Rg. 23; Rg. 25 shows 
the help-agent activities to be outputted in the cor- 
responding help-agent program number in the table 
of Rg. 24; Rg. 26 is a flowchart of the main routine 
of the agent control operation in accordance with 
this embodiment; Rgs. 27A-27C Illustrate an exam- 
ple of the agent control operation made by tiie f tow- 
chart of Rg. 26; Rg. 28 shows contents of the agent 
data menrK>ry in still another modified embodiment; 
Rg. 29 is an example of tiie sub-agent program 
table in the agent data memory of Rg. 28; Rg. 30 
shows short-cut agent activities to be outputted in 
the corresponding short-cut agent program number 
in tiie table of Rg. 29; Fig. 31 is an example of the 
contents of the frequent program data in the agent 
data m&nory of Rg. 28; Rg. 32 is a flowchart of the 
main routine of the agent control operation in 
accordance with this emtxxiiment; Rg. 33 is an 
explanatory view showing multi-stage operation of 
the communication program for menu selection; 
Rg. 34 is an explanatory view showing the short-cut 
agent activity incorporated into the mufti-stage 
operation of Rg. 33; Fig. 35 is a flowchart of the 
short-cut agent operation that is a sub-routine 
branched from the main routine of Fig. 15; Rg. 36 
shows an example of the display when tiie short-cut 
agent appears thereon together with the main 
agent; Fig. 37 shows an example of the display 
when only the main agent appears therieon; Fig. 38 
shows contents of the agent data memory in still 
another nmlified emtxxiiment; Fig. 39 shows an 
example of the study item data In the agent data 
memory of Rg. 38; Rg. 40 shows an example of the 
background select table in the agent data memory 
of Rg. 38; Rgs. 41A-41D show examples of tiie 
selected t>ackground that is shown on tiie display 
together witii the prescribed agent's activity; Rg. 42 
shows an example of tiie response recognition data 
in the agent data memory of Fig. 38; Rg. 43 is a 
flowchart of the agent control operation to be exe- 
cuted in this embodiment as a sub-routine 
branched off from the main routine of Rg. 15; Rgs, 
44B and 44C show pictures successively appearing 
on the display on the conditions of Rg. 44A in this 
embodiment; Rg. 45 shows contents of the agent 
data memory in still another modified embodiment; 
Rg. 46 shows an exanple of the appearance select 
table In the agent data memory in Fig. 45; Rg. 47 is 
a flowchart of the agent control operation to be exe- 
cuted in tills embodiment as a sub-routine 
branched off from the main routine of Rg. 15; Figs. 
48A-48C show an example of tiie agent's appear- 
ance and activity appearing on the display in this 
emtxxiiment; Rgs. 49A-49B show another example 
of the agenfs appearance and activity in tiiis 
emtxxJiment; and Rg. 50 show still another exam- 
ple of ttie agent's appearance and activity in a mod- 
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ified enixxliment. 

The first embodiment of the present invention will 
be hereinbelow described In more detail in reference to 
Rgs. M 1 of the accompanying drawings. 

In this embodiment a personified agent appears on 
a graphic display In a vehicle. The agent may be repre- 
sented as a plane image or threeKlimenstona) image by 
holography, for example. The activity (inducing utter- 
ance) of the agent is controlled in response to the vehi* 
cle condition and the result of past record study. The 
vehicle condition may include various condition of the 
vehicle itself, the driver, the passenger(s) and preceding 
and succeeding vehicles. The past record study means 
study of past record of the vehicle condition and may 
include reaction of the user (driver) to the agent* s activ- 
ity appearing on the display. Thus, the driver may com- 
municates with the agent in the vehicle, even while 
driving. 

The "agent" used herein is defined as an imaginary 
personified subject who can make judgement and study 
like a human body. Accordingly, the same agent makes 
different communication with the driver on the different 
vehicle condition. Even on the same vehicle condition, 
the past record study may result in different communica- 
tion by the agent. The agent Is not a superhuman and 
may sometime make mis-judgement and blunder as fas 
as it does not affect the vehicle driving. Such mis-judge- 
ment and blunder made by the agent may be conected 
in accordance with the driver's reaction, which is a kind 
of the past record study. 

The agent represented on a display may be like a 
human body. It may also be like an animal such as a 
chick, dog. cat, hog, rat. etc. it may also be a robot or 
any character The agent may be a chikl at the begin- 
ning and is grown with time. The appearance and voice 
of the agent may be selected from predetermined plural 
ones. 

Fig. 1 is a block diagram of the device embodying 
the present invention. There Is a processing unit 1 
including a navigation processing unit 10 for finding out 
a drive route to a predetermined goal ttiat is known to 
the driver by visual and/or voice message, an agent 
processing unit 11 for controlling the activity of the 
agent an interface (1/F) 12, an Image processing unit 
13. a voice control unit 14 and a current status data 
processing section 15. 

Navigation processing unit 10 and agent process- 
ing unit 11 have respectively a central processing unit 
(CPU) tiiat operates in response to the input information 
to perfonn various data processing operation and con- 
trol of parts. To CPU are connected ROM, RAM and 
timer through data bus lines. Navigation processing unit 
10 and agent processing unit 11 are connected with 
each other by a network so that data processed in one 
unit may be given to the other unit ROM is a read-only- 
memory storing various data and programs necessary 
to perform control in CPU. RAM is a random-aocess- 



memory to be used as a working memory when CPU 
performs data processing operation. 

CPU of navigation processing unit 10 ond agent 
processing unit 1 1 reads out tfie programs in ROM to 

5 conduct necessary operation. In modified embodiment. 
CPU reads out the programs from a memory medium in 
a memory drive 23, in which case the programs are then 
installed in an agent data memory 29. a navigation data 
memory 30, a hard disk (not shown) or any other mem- 

10 ory. A necessary program is loaded from the memory to 
RAM for execution. It may be possible that CPU reads 
out ttie necessary program from the menK)ry drive 23 to 
load it directly to RAM. 

To navigation processing unit 10 are connected a 

IS current position sensor 21 and navigation data memory 
30. To agent processing unit 1 1 is connected agent data 
memory 29. An input device 22, memory drive 23 and a 
communication control device 24 are connected to l/F 
12. A display 27 and a camera 28 are connected to 

20 image processing unit 13. A voice processing unit 25 
and a miaophone 26 are connected to voice control unit 
14. A current status sensors 40 are connected to cur- 
rent status data processing section 15. 

Current position sensor 21 for detecting the current 

25 absolute position (its latitude and longitude) of the vehi- 
cle has a GPS (global positioning system) receiver 211, 
a direction sensor 212. a steering sensor 213. a dis- 
tance sensor 214 and a beacon sensor 215 that 
receives tine position signal from, a beacon located on 

30 tiie road. GPS receiver 21 1 receives radio waves from 
earth satellites to determine the vehicle position. Bea- 
con sensor 215 receives positional information from 
beacons anranged on roads. GPS receiver 21 1 and bea- 
con sensor 215 can solely act to measure the vehicle 

35 position. Where GPS receiver 21 1 can not receive the 
radio waves and beacon sensor 21 5 can not receive the 
positional information, direction sensor 212 and dis- 
tance sensor 214 cooperates with each other to infer 
ttie current position. 

40 Direction sensor 212 includes a terrestrial magnet- 
ism sensor that detects tenesti^ial magnetism to deter- 
mine orientation of the vehicle, a gyromagnetic sensor 
tiiat detects a rotational angle velocity of tiie vehicle that 
is integrated to determine orientation of the vehicle, a 

45 pair of wheel sensors that detect an output pulse differ- 
ence (a difference of movement between left and right 
wheels) to detect an angle of gyration of the vehicle. 
Steering sensor 213 employs an optical sensor or rota- 
tion resistance volume mounted to a rotating member of 

so a steering to detect a steering angle. Distance sensor 
214 detects a distance of movement by detecting revo- 
lution of the wheels or acceleration, for example. 

Input device 22 is used to input any data, for exam- 
ple, the cun-ent position and the goal when starting nav- 

$5 igation processing. Input device 22 also act as means 
for inputting tiie driver's response to the agent's mes- 
sage or question. An example of input device 22 is a 
touch panel arranged on display 27 and adapted to 
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input any information by a touch on a Key or menu rep- 
resented thereon. Another example of Input device 22 is 
a keyboard, mouse, bar code reader, write pen, joy 
sticK infrared remote controllable input device and voice 
analyzer. An infrared remote controllable input device 
may cooperate with a receiver that receives signals 
from the input device. The remote controllable input 
device has a joy stick for moving a cursor on a display 
section, menu designating keys or buttons and ten keys. 

Memory drive 23 drives memory medium storing 
the computer program that is required to perform oper- 
ation of navigation processing unit 10 and agent 
processing unit 1 1 . The computer program stored in the 
memory medium includes various programs and data. 
The memory medium vns^ be any one capak)le of stor- 
ing the computer program. For exarr^e, magnetic 
memory medium such as a floppy disk, hard disk and 
magnetic tape, semi-conductor memory medium such 
as a memory tip and IC card, magneto-optical memory 
medium such as CD-ROM, MO and PD, etc. may be 
examples of suitable memory medium. Any printed mat- 
ter desaibing the computer program may also used In 
cooperation with a character analyzer. 

When a rewritable memory medium such as a 
floppy disk and IC card is used, it may also be used to 
store data in RAM of navigation processing unit 10 and 
agent processing unit 1 1 or memory 29, 30. For exam- 
ple, study data (study item data and response data) 
regarding activities of the agent is stored in an IC card. 
The driver may use the IC card storing tiie driver's own 
data in common for two or more vehicles, as far as these 
vehicles have the same device. This means that tiie 
agent is peculiar not to tiie vehicle but to the driver. 

Communication control device 24 transmits and 
receives data to and from a portable telephone and per- 
mits communication between the driver and his or her 
friend outside of tiie vehicle. Communication control 
device 24 also receives data regarding road information 
including traffic jams and traffic acctd^ information 
that are supplied from a traffic infbrmation center ft also 
receives Karaoke data. The study data regarding activi- 
ties of tiie agent may be received and transmitted 
through communication control device 24. 

Voice output device 25 comprises a plurality of 
speakers arranged in the vehicle through whk:h a voice 
controlled by voice control unit 14 is outputted. The out- 
putted voice includes a voice message for guidance of 
the drive route and the agent's voice or sound that is a 
kind of tiie agent's activities. The speakers for audio 
instruments may be used as voice output device 25. 
Voice control unit 1 4 changes accent and tone quality of 
the voice outputted by voice output device 25 in 
response to tiie driver's tuning oonrvnand. 

A microphone 26 functions as voice input device for 
inputting any voice which is then processed by voice 
control unit 14. For example, a voice for inputting the 
goal of the drive route in navigation processing opera- 
tion and the driver's voice in response to the agent's 



activities are inputted through miaophone 26. A micro- 
phone for Karaoke may be used as microphone 26. 
However, microphone 26 shoukl preferably have direc- 
tionality enough to collect the driver's voice. A hand-free 

5 unit conprising speaker 25 and microphone 26 nnay be 
used for telecommunication. 

Display 27 used to represent the results of oper- 
ation by navigation processing unit 10, for example, 
guidance to operation, operation menu, operation keys, 

10 a recommended drive route to tiie goal determined by 
the user's input through input device 22. and a guide 
map ak>ng ttie drive route. Display 27 also represents 
the results of operation by agent processing unit 1 1 , for 
example, the activities of the agent. Any image taken by 

15 camera 28 is also shown on display 27 after being proc- 
essed by image processing unit 13. As display 27 may 
be used a CRT display, a liquid crystal display, a plasma 
display or a hologram device that projects a hologram 
onto a front glass. 

20 Camera 28 conprises CCD cameras that take pho- 
tographs inside and outside of the vehicle. An inside 
CCD camera mounted in the vehicle take photographs 
of tiie driver. CCD cameras are also mounted at tiie 
front, rear, right side and left side for taking photographs 

2s of surroundings of the vehicle. The photographs taken 
by camera 28 are supplied to image processing unit 1 3 
for image analyzing operation. The result of tiie image 
analyzing operation by image processing unit 13 is used 
to determine a program number by agent processing 

30 urvtll. 

Agent data memory 29 stores data and program 
necessary for agent processing operation in accord- 
ance witfi this embodiment of the present invention. 
Memory 29 comprises memory medium such as a 

35 floppy disk, hard disK CD-ROM, magneto-optical disk, 
magnetic tape, IC card, magneto-optical card. etc. and 
a drive for tiie memory medium used. In a preferred 
embodiment, study item data 292 and response data 
293 are stored in a portable memory medium such as 

40 IC card and ftoppy disk, and ottier data is stored in a 
hard disk. 

Agent data memory 29 stores agent program 290. 
program select table 291. study item data 292. 
response data 293 and image data 294. Study item data 

45 292 and response data 293 represents the result of 
study of tiie agent through ttie driver's operation and/or 
response. Accordingly, study item data 292 and 
response data 293 are peculiar to each driver and 
renewed when the driver makes some operation and/or 

50 response. 

Image data 294 stores tiie agent appearing oh dis- 
play 27. As described before, the agent represented on 
display 27 may be like a human body, like an animal 
such as a chicK dog. cat. frog, rat. etc.. rolTOt or any 

55 imaginary character. The agent may be a child at tiie 
beginning, which is grown witti time as a result of study 
Image data 294 stores a plurality of ttie agent appear- 
ance, one of which may be selected by the driver 
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through input device 22. 

Agent program 290 stores agent processing pro- 
gram and communication program in numerical order. 
Agent |»'Ocessing program oommarvis activities of the 
agent Communication program commands communi- 
cation between the agent and the driver which is output- 
ted through display 27 and speaker 25. 

Agent program 290 stores plural kinds of voice pat- 
terns, for example. man*s voice, woman's voice, child's 
voice, mechanical sound, animal voice, famous actor's 
or actress's voice, and popular character's voica The 
driver can select fevorite one of the voice patterns 
stored in agent program 290 through input device 22. 

Program select table 291 is a table for use in selec- 
tion of a specific one from communication programs 
stored in agent program 290. An example of program 
select table 291 is shown iri Fig. 2. Once a specific pro- 
gram is selected in reference to program select table 
291 In Fig. 2, the same program number is applied to an 
activity pattern select table, an example of which is 
shown in Fig. 3, to determine the activity (appearance 
and voice) of the agent The program number of tables 
in Figs. 2 and 3 are identical to the communication pro- 
gram number stored in agent program 290. 

Rgs. 4A-4D show an example of the agent activi- 
ties, which is shown on display 27 when one of the pro- 
gram Nos, 00001*00006 is selected (see Fig. 3). This 
activity start with the image of Fig. 4A which is changed 
to the images of Figs. 4B and 4C. then to the image of 
Fig. 4D that is the same as the starting image of Fig. 4A. 
This activity is a respectful and ceremonious bow, 
because this is almost the first time that Lisa meets the 
driver. The name "Lisa" is only an example and any 
name may be inputted by tiie driver. With the agent's 
action of obeisance appearing on display 27. the agents 
says sometiiing that the driver can hear through 
speaker 25. In the program No. 00001 or 00002, the 
agent makes an obeisance while saying "Nice to meet 
you, I'm Lisa I wish to go with you", which is. however, 
changed depending upon tiie vehicle's and driver's cur- 
rent status and as a result of ttie past record study. 

In this embodiment, input device is also used to 
input the user-related infomnation 298 to be stored in 
agent data memory 29a. The user-related infomiation 
298 may include the user's age, sex, hobby, character, 
favorite regarding the agent, etc. The user-related infor- 
mation 298 stored in agent data memory 29a is read out 
by agent processing unit 1 1 to determine the personal 
characteristic of the agent. The user-related information 
298 is also used to determine utterance pattern of the 
agent to be outputted through voice output device or 
speaker 25. The user-related information 298 may be 
obtained from tiie user's response to the agenfs ques- 
tion, which is inputted through microphone 25. 

When cun'ent status sensor 40 detects that the 
coolant temperature is lower than a predetermined 
level, the agent shows a sleepy activity. For example, 
the agent gives or stifle a yawn before making a prede- 



termined action (ag., bow). In another example, the 
agent shows a slower movement and speaking. Such 
sleepy expression may be changed as a result of tiie 
study A cheerful activity may be expressed by the agent 
5 who speaks loudly, appears on display while running, 
for example. 

The respective items shown on the abscissa of 
table 291 of Fig. 2 determine the program number, on 
tiie ordinate, to be selected. The items on ttie absdssa 

10 may be classified into two groups, one being deter- 
mined from the vehicle or driver's condition detected by 
current status sensor 40 which includes time, vehicle 
location where ignited, coolant temperature, etc. and 
the otiier being determined from the study results 

IS stored in study item data 292 and response data 293 
. which includes the number of today's ignition-on opera- 
tion, time passage from the last ignition-off operation, 
the total number of ignition, etc. In table 291 of Rg. 2. a 
circle indicates the item that is indispensat)le to select 

20 the program of a corresponding number. A hyphened 
item is not considered in selecting tiie program of a cor- 
responding number. 

Altiiough Figs. 2 andi 3 describes tiie activity pat- 
tern (bow) to be represented in response to the igniting 

25 operation, various activity pattems are likewise defined 
in response to some operation. For example, push of a 
brake pedal to an extent greater tiian a predetermined 
level is regarded as a key operation, in response to 
which the agent may fall on his or her behind, take sev- 

30 eral steps backward, keep his or her foot cries in sur- 
prise, etc. These responsive activities may be changed 
by the agent's study of sudden braking operation. For 
example, tiie program determines that tiie agent falls on 
his or her behind at the first to tiiird times of sudden 

35 braking operation, takes several steps backward at the 
fourth to tenth times, and keeps his or her foot thereaf- 
ter. Such program represents that the agent becomes 
experienced and accustomed to the driver's sudden 
braking operation. It may be tiiat when a time of more 

40 tiian one week has been passed since the last time sud- 
den braking operation, the agent's activity of one eariier 
stage is applied. For example, where the fiftii time sud- 
den braking operation occurs ten days after tiie last time 
(the fourth time) operation, the program, determines 

45 such that the agent falls on his or her behind. 

Study item data 292 and response 293 are stored 
and renewed by the agent's study, examples of which 
are shown in Figs. 5 and 6, respectively Study item data 
292 shown in Fig. 5 includes the total number of igni- 

50 tion-on. ttie last ignition-off. today's ignition-on, the fuel 
remainder record, the skip record, the default value, etc. 

The total number of ignition-on operation is 
increased by one each time the ignition switch is turned 
on to ignite the engine. Each time the ignition switch is 

55 turned off. its date and time is renewed at tiie last igni- 
tion-off data. Each time the ignrtion-on operation occurs, 
its numt>er (in a day) and its ignita'on-off time are 
recorded in today's ignition-on data. When a day is over 
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(at 24:00), this data is cleared to zero, awaiting the next 
ignition-on operation. TTie time for renewing the today's 
ignition-on operation data is normally 24XK). but may be 
changed upon the driver's demand. 

The fuel remainder record stores fuel remainders at 
the last five times ffll-up. Each time the fuel is refilled, 
the newest data Is stored at the rightmost column, and 
the last four data are moved to the adjacent left col- 
umns, respectively The oldest data in the leftmost col- 
umn is deleted. From the last five fuel remainders may 
be obtained the average fuel remainder (G2). When the 
fuel sensor 415 detects tiie current fuel remainder (01) 
is below the average fuel remainder (G2). the agent 
appears on display 27 to show a f ill-up suggesting activ- 
ity and says "IVe got hungry. I want fuel." through 
speaker 25. 

The skip record data is counted up each time the 
driver refuses or neglects the agent's recommendation 
or communication even when such program nuniser 
has been selected under control of the present inven- 
tion. The study item data 292 includes the skip record 
where the agent activities, of ttie program number 
involves the driver's response, such as by Program No. 
00123 wherein the agent suggests stop of operation of 
the air-conditioner. 

The default value includes initial values such as 
time, number, tenperature. vehicle speed, data, etc. 
The default value is used to initialize data when data 
has been changed by the driver, for example, when the 
driver has changed the ignition-on operation time from 
the initial value of 24:00 to his or her desired time, as 
described before. 

The otiier data tiiat may be included in ttie study 
item data 292 include, for example, birthdays of the 
driver and his or her relatives and friends (that can be 
inputted by the driver), national holidays and their histo- 
ries, date of events such as Christmas Eve, St Valen- 
tine Day, etc. Amorrg the communication programs, 
there may be special menu for special day In accord- 
ance with an example of the special menu communica- 
tion program, the agent is appearing on display 27 while 
dressed as Santa Glaus on Christmas Eve and Day 

The response data 293 in Rg. 6 stores the user's 
response history to the agent's activities for the commu- 
nication programs which irtdude the user's response as 
one of the study data. The response data have different 
contents for different communication programs. In an 
example shown in Fig. 6A, the response data for the 
programs of Nos. 00123 and 00125 store several latest 
data (time and content). The response data of Program 
No. 00124 stores tiie only one latest response content, 
which is renewed each time the driver makes a 
response. In other examples, the response data may be 
several latest response contents (not including 
response time), only one latest response data of time 
and content, only one or several latest response date 
(not including response contents, etc. 

The symbols A. B and 0 in Rg. 6A indicate the 



response contents. As shown in Rg. 6B, the syni)ol A 
indicates "Neglected". B "Refused" and C "Accepted". 
The driver's response content may be discriminated by 
analyze of tiie driver's voice inputted through micro- 
5 phone 26 or of information inputted ttirough input device 
22. 

Although in this embodiment the driver's response 

contents are classified into three patterns, "Neglected" 
(A). "Refused" (B) and "Accepted" (C). it may have more 
10 detailed classification. For example. "Resolutely 
Refused". "Angrily Refused". "Pleased", etc. may be 
added. 

Rg. 7 represents data files stored in navigation data 
memory 30 (Rg. 1). As shown in Rg. 7, navigation data 

IS memory 30 stores communication area data file 301. 
map data file 302. crossing data file 303. node data file 
304. road data file 305. route-finding data file 306 and 
photo data file 307. Navigation data memory 30 com- 
prises one or combination of storage medium including 

20 a floppy disk, hard disk, CD-ROM. magneto-optical disk, 
magnetic tape. IC card, magneto-optical card. etc.. and 
drive units tiierefor. For example, route-finding data file 
46 is made of rewritable memory such as flush memory 
while other data files are stored in CD-ROM. and drive 

25 units therefor are respectively used. 

Communication area data file 301 stores communi- 
cation are data per handy phone types. Communication 
data is used for representation, on display 27, of com- 
municatable area by tiie driver's handy phone con- 

30 nected or non-connected to communication confrol unit 
24. This data is also used while finding out the drive 
route to the goal. The respective communication area 
data is assigned its peculiar number for easy access 
tiiereto. The communicative area is defined by continu- 
es ity of short segments, so that the communicative area 
data may be specified by a plurality of coordinate data 
representing points connecting the segments. In 
another example, the communicative area is divided 
into plural rectangles, each rectangles being defined by 

40 two diagonal points thereof, in which case the communi- 
cative area data comprises a plurality of coordinate data 
representing diagonal points of the respective rectan- 
gles in the communicative area. 

Data in communication area data file 301 is prefer- 

45 ably renewable to conform with change of the communi-r 
cative area of the handy phone. To meet this 
requirement, the handy phone is connected to commu- 
nication control unit 24 for communication witii an infor- 
mation center for updating of the contents of 

so communication area data file 301 . In another example, 
communication area data file 301 is made of rewritable 
and updatable memory such as a floppy disk and IC 
card. 

Map data ffle 302 stores map data to be repre- 
ss sented on display 27. Map data is stored in a tree con- 
struction. The uppermost layer stores map data of the 
broadest territory, and lower layer stores more detailed 
map data of smaller territories. The map data of the 
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respective layers are assigned their peculiar map 
codes. 

Crossing data file 303 stores crossing data such as 
crossing numbers spedfying the crossing one by one. 
crossing names, crossing coordinates, road specifying 
numbers start or terminate with the crossings, existence 
of signals at the crossings, etc. 

Node data file 304 stores node data specifying 
coordinates of respective points on the respective 
roads. Any road section may be defined by a plurality of 
node data thereon between the beginning and end of 
the section. 

Road data file 305 stores road nuni^ers specifying 
the road one by one. the crossing numbers at the begin- 
ning and end of the respective roads, other road num- 
bers of the road that begins or terminates with the same 
point, road width, prohibition information (no parking, no 
entrance, one-way road. etc.). number of photo data, 
etc. 

Data stored in crossing data file 303. node data file 
304 and road data file 305 are particularly used for 
route-finding. 

Route-finding data file 306 stores a series of cross- 
ing data and a series of node data which constitute the 
route found in the navigation system. A series of cross- 
ing data includes data regarding names and numbers of 
the crossings on the suggested drive route, numbers of 
the photos showing these crossings, names and num- 
bers of the crossings at which the vehicle should turn to 
follow the suggested drive route, distances between the 
crossings, etc. A series of node data includes data 
regarding coordinates of the respective nodes on the 
suggested drive route. 

Photo data file 307 stores photos showing the 
crossings, conspicuous landscape and buildings, for 
example, along the suggested drive route. Photos are 
assigned their peculiar photo numbers and stored in the 
form of digital, analog or negative films. 

As shown in Fig. 8. cunrent status sensor 40 
includes an ignition sensor 401 , a vehicle speed sensor 
402. an accelerator sensor 403, a brake sensor 404, a 
side brake sensor 405, a shift position sensor 406, a 
winker sensor 407. a wiper sensor 408. a light sensor 
409, a seat-belt sensor 410, a door opening sensor 41 1 . 
a passenger sensor 412, a room temperature sensor 
41 3, an open air temperature sensor 41 4. a fuel remain- 
der sensor 415. a coolant temperature sensor 416, an 
ABS sensor 417, an air-conditioner sensor 418, a 
weight sensor 419. a fore-distance sensor 420, a rear- 
distance sensor 421 . a body temperature sensor 422, a 
pulse rate sensor 423. a perspiration sensor 424. a 
brain wave sensor 425. an eye-tracer 426, an infrared 
sensor 427. and other sensors 428 for detecting, for 
exanpie. decrease of air pressure of tires, loosening of 
belts, opening of windows, operation of horn, oil temper- 
ature, oil pressure, etc. As noted from the above, cur- 
rent status sensor 40 is used to detect various vehicle 
conditions and driver's conditions in this embodiment In 



nrxxlification, only vehicle conditions or only driver's 
conditions may be detected by current status sensor 40. 
Sensors are mounted at suitable location inside or out- 
side of the vehicle, respectively. Usually one sensor is 

5 used for one sensing purpose. However, it may be pos- 
sible that some kind of sensor receives tiie detection 
signal from another sensor to carry out indirect sensing. 
For example, an air-pressure sensor may indirectfy 
detect a decrease of an air-pressure of a tire in 

10 response to a change of a signal from a wheel vefodty 
sensor. 

Ignition sensor 401 detects ON and OFF of an igni- 
tion starter. Vehicle speed sensor 402 may be of any 
conventional type Including one wherein a rotation 

IS angle velocity or revolution of a speed meter cable is 
detected to calculate a vehicle speed. Accelerator sen- 
sor 403 detects a degree of push stroke of an accelera- 
tor pedal. A brake sensor 404 detects a push stroke, 
push force or pt^h speed of a brake pedal to discrimi- 

20 nate if the driver makes an emergency brake. Side 
brake sensor 405 detects operation or non-operation of 
a side brake. Shift position sensor 406 detects the cur- 
rent position of a transmission shift lever. Winker sensor 
407 detects which winker is lightened. Wiper sensor 

25 408 detects wiper drive conditions (espedally. wiper 
speed). Light sensor 409 detects various lights and 
lamps such as head lights, tail lamps, fog lamps, room 
lamps, etc. being lightened. Seat-belt sensor 410 
detects if the driver and passengers are equipped witti 

30 seat-belts. If at least one of ttiem is not equipped with a 
seat-belt, the corresponding communication program is 
working such tiiat ttie agent appears on display 27 to 
give warning, notice, advice, etc.. one of which is 
selected by the results of study 

35 Door opening sensor 41 1 detects opening of vehi- 
cle doors. When detecting a half opening of the door, it 
is informed to the driver or passenger by the agent's 
activities (action and voice). Door opening sensor 411 
may comprise plural sensors each detecting opening of 

40 doors at the drivers seat, at the assistant driver's seat, 
at opposite rear seats, etc.. respectively Passenger 
sensor 412 detects if there are any passengers In the 
vehicle, by the photo showing the interior of the vehicle 
taken by camera 28 or by pressure sensor or weight 

45 sensor mounted under the seat. Room temperature 
sensor 413 detects a room temperature of the vehicle, 
and open air temperature sensor 41 4 detects a tenrper- 
ature outside of the vehicle. 

Fuel remainder sensor 415 detects a remainder in a 

so fuel tank. As described before, the communication pro- 
gram Is selected when the fuel remainder is decreased 
to the average of the last five records, so that tiie agent 
urges the driver to refill the fuel. 

Coolant temperature sensor 416 detects a coolant 

55 temperature. When detecting tiiat tiie coolant tempera- 
ture is lower than a predetermined level at the time 
immediately after the ignition switch Is turned on. tiie 
agent usually takes a sleepy action. On the otiier hand, 
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when the coolant tempOTture is higher than another 
predetermined level, such communication program is 
selected that the agent looks weary and gives warning 
or notice to the driver to prevent overheating of the 
engine. 

ABS sensor 417 detects if an ABS (anti-braking 
system) is in cperativa As well known. ABS prevents 
the tires from being locked due to emergency braking to 
thereby improve drivablility and stability of the vehicle. 

Air-conditioner sensor 41 8 detects the conditions of 
an air-conditioner mounted in the vehicle, including ON 
and OFF thereof, the set temperature and the set wind 
volume, etc. Weight sensor 419 detects the driver's 
weight, which is used solely or in combination with the 
photos by camera 28 to specify ttie driver so that the 
agent's activities can be determined in accordance with 
study item data 292 and response data 293 inherent to 
the said driver. When there are different agents selecta- 
ble by plural drivers, one specific agent is determined by 
identification of tiie cun'ent driver. 

Fore-distance sensor 420 detects a distance to 
anotiier vehicle or obstacles ahead of the vehicle cur- 
rent position on the road, whereas rear-distance sensor 
421 detects a distance to anotiier vehicle or obstacles 
astern of the vehicle cun'ent position on the road. 

Body temperature sensor 422. pulse rate sensor 
423 and perspiration sensor 424 detects a body temper- 
ature, a pulse rate and perspiration of the driver, respec- 
tively These sensors may be mounted on ttie surface of 
a steering to detect conditions of the driver's hand. Body 
temperature sensor 422 may be a thermography utiliz- 
ing infrared sensor elements to detect tiie driver's body 
temperature distribution. Brain wave sensor 425 detects 
alpha wave or beta wave of tiie driver to discriminate the 
degree of awakening of tiie driver. Eye-tracer traces eye 
movement of tiie driver to tiie driver's intention and 
awakening. Infrared sensor 427 detects movement of 
the user's hands and head. 

In reference to the flowchart of Fig. 9 showing the 
main routine of the agent control operation of tiie device 
of this embodiment, agent processing unit 1 1 first initial- 
izes the deviceafter ignition sensor 401 detects that the 
ignition switch is turned on. at step S1 1 . Initialization 
includes clearing of RAM, allocation of ttie respective 
woridng areas in RAM. loading of program select table 
291 (Fig. 2} into RAM. set of flags to zero, etc. Afthough 
the operation starts with turning of the ignition switch in 
this embodiment, it may start in response to detection of 
opening or closure of any door by door opening sensor 
411. 

Next, agent processing unit 1 1 receives the current 
status data, stores them in ttieir predetermined areas in 
RAM and analyzes ttiem to confirm the current vehicle 
and user's status, at step 812. The cunrent status data 
include data detected by the various sensors in current 
status sensor 40 and supplied therefrom to agent 
processing unit 1 1 through current status data process- 
ing section 15. The image data obtained by camera 28 



is processed by image data processing section 13, 
which is also inputted to agent processing unit 1 1 . The 
current vehicle position data detected by current vehicle 
position sensor 21 is also inputted to agent processing . 

5 unit 1 1 through navigation processing unit 10. By way of 
example of the current status data, where coolant tem- 
perature sensor 416 detects a coolant temperature t1. 
agent processing unit 1 1 stores this temperature t1 in a 
predetermined area in RAM and. if the temperature t1 is 

10 below a predetermined temperature t2. disaiminates 
tiiat tiie coolant temperature is low (see the coolant 
temperature column in table of Rg. 2). 

When the driver inputs any request through micro- 
phone 26, this is processed by agent processing unit 1 1 

15 as a kind of the current status data. Such driver's 
request may be inputted by saying "Phone to Mr. Thin- 
gummy", "Show me restraints around here". "Play a 
CD-, etc. The driver's request is processed by voice 
processing unit 14 with keywords such as "phone", 

20 "restraint", "play", "CD". These keywords are arranged 
on the abscissa of program select table 291 in Fig. 2. 

Agent processing unit 11 then checked study data 
(study item data 292 and response data 293) in agent 
data memory 29 to confirm the agent's study regarding 

25 the obtained current status data, at step S1 3. 

From tiie obtained current status data and tiie study 
data thereof, agent processing unit 11 discriminates if 
there is any communication program to be initiated, in 
reference to program select table 291 , at step 814. If not 

30 (No at 814), the procedure is returned to step 812 
where it awaits another current status data. Where 
tiiere is a communication program to be initiated (Yes at 
S14), it confirms the program number tiiereof. It further 
reads out the driver's response record stored in 

35 response data 293 for tiie said program number to 
tiiereby determine if the communication program of tiie 
selected program number should be initiated or sus- 
pended, at step SI 5. 

If tiie answer at SI 5 is No, agent processing unit 1 1 

40 operates to initiate ttie selected communication pro- 
gram so that the predetermined agent's activities are 
shown on display 27 and outputted tiirough speaker 25, 
at step 816. This enables good communication 
between the driver and the agent which reflects not only 

45 ttie vehicle and driver's current status but also ttie result 
of study of ttie past status data and ttie driver's 
response records. 

When the driver shows any response to the agent's 
activities made at 816, such response data is obtained 

50 ttirough microphone 26 or input device 22. at step 81 7. 
Then, agent processing unit 1 1 disaiminates if the 
communication program made at 816 involves any sug- 
gestion, proposal or question that awaits ttie driver's 
. response or answer. For example, agent says "Do you 

55 want to hear any radio program ?" or "Do you feel hun- 
gry? May I find any restraint near here?". The latter pro- 
gram may be initiated in response to detection of a 
hungry sound through microphone 26. Such a kind of 
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communication program is hereihiater refenred to as a 
'questioning program**. 

When the selected communication program is not a 
questioning program (No at S18), the procedure is 
advanced to a step S20. When it Is a questioning pro- 
gram (Yet at Si 8). at a step Si 9. agent processing unit 
1 1 controls the agents activities in accordance with the 
driver's response already obtained at 81 7. For example, 
when the driver indicates the intention to accept the 
agent's proposal "Do you want to hear any radio pro- 
gram ?" by answering "Yes", "OK", "Please", eta, agent 
processing unit 11 urges the agent to act accordingly 
and turns the radio receiver on to follow the driver's 
request. 

Agent processing unit 1 1 accumulates data regard- 
ing the execution of tiie selected communication pro- 
gram to obtain the study data, at step S20. For example, 
when the selected communication program is not exe- 
cuted (Yes at 815). the number of skip of this program 
number stored in study item data 292 is increased by 
one. When the cunrent number of skip is lower by one 
than a predetermined maximum number, all records 
now stored in study item data 292 and response data 
293 for that program number should be cleared. Except 
this case, tiie cun'ent record In the study item data 292 
is renewed when tiie current status data acquired at 
Step 12 includes some study Item, and the response 
data obtained at Step 1 7 is stored in the response data 
293 when the selected communication program is a 
questioning program tiiat requests tiie driver's 
response. When a predetermined number of response 
records have already been stored in the response data 
293, the oldest response data is disused. 

After executing the above-desaibed steps, at step 
821, agent processing unit 11 confirm tiiat a flag has 
become one in a flag area that was set in RAM at Step 
11. When tiie flag is still zero (No at 821). it then dis- 
criminates If ignition sensor 401 detects that the ignition 
switch is turned off. at step 822). When not detected 
(No at 822), tiie procedure is returned to Si 2. When 
turning-off of the ignition switch is detected (Yes at 822). 
the flag in the flag area In RAM is set to one. at step 
823. and ttien tiie procedure is returned to 812. When 
the flag is set to one, agent processing unit 1 1 controls 
such that the agent takes a farewell action that is the 
final activity in the selected communication program (at 
816). and obtains the study data (at 820). It also exe- 
cute the automatic extinguish any light that the driver 
has forgotten to turn off. Then, the control procedure 
terminates (Yes at 821). 

Figs. 1 0A-IOC show an example of tiie agent activ- 
ities that is controlled by agent processing unit 1 1 after 
the ignition switch is turned on. In tiiis example. Ignition 
sensor 401 detects that the ignition switch is on. current 
position sensor 21 detects tiie current vehicle position is 
other than the driver's home and rest positions. These 
data are supplied to agent processing unit 1 1 as tiie cur- 
rent status data (at 81 2). Regarding study item data 292 



and response data 293. it confirms that this is the sec- 
ond ignition-on operation today, the last agent control 
operation was terminated at 10.-05, on July 12. 1997, 
and the total number of ignition-on operation amounts to 

5 30 times, at 813. 

From ttiese data, agent processing unit 11 deter- 
mines the communication program of No. 00004 should 
be executed, in reference to the program select table 
291 shown in Rg. 2 (at 814). The agent activity to be 

10 outputted through display 27 and voice output device 25 
is an "obeisance" that is a respectful bow. witii saying 
"Hello. I'm Lisa. I will go with you again". After oonfimv 
ing tiiat this communication program should not be sus- 
pended (No at 81 5). agent processing unit 1 1 starts this 

IS communication program (at S16). as shown in Fig. 10B. 
The agent's obeisance is represented by a series of sta- 
tionary Images or animation. 

Since tiie program of No. 00004 is not a questioning 
program that awaits the driver's response (No at 818). 

20 the study data is stored and renewed as a result of exe- 
cution of the program 'of No. 00004 (at 820). In this 
example, tiie number of today's ignition-on operation is 
changed from the previous value "2" to "3", and tiie total 
number of ignition-on operation is changed from the 

25 previous value "30" to "31 ". Since tiie flag is still zero at 
tills time (No at 821) and the ignition is still on (No at 
822). tiie procedure is returned to 812 so that this com- 
munication program continues. 

Since tiie total number of ignition-on operation has 

30 reached "31", the communication programs of t^. 
00001-00006 can not be selected (see table 291 in Rg. 
2), as far as all study data (study item data 292 and 
response data 293) are initialized. Until tiie total number 
of ignition-on operation reaches "300", one of the pro- 

35 grams of Nos. 00007-00019 is selected In response to 
tiie current status data. The agent no more makes an 
"obeisance". The agent makes an ordinary "Ixwv" when 
the coolant temperature Is high, while a "sleepy tx)w" is 
applied when tiie coolant temperature is low. The 

40 agenfs voice depends on tiie program number to be 
selected by data such as the place where the ignition 
switch Is turned on, the number of today's ignition-on 
operation, etc. 

Rgs. 11A-11C illustrate another example of tiie 

45 agenfs activities that may be selected during driving. In 
tills example, air-conditioner sensor 418 detects tiiat 
tiie air-conditioner is In operation, and temperature sen- 
sors 413. 414 detects a room temperature T1 and an 
open air temperature T2, respectively. These data are 

so supplied to agent processing unit 1 1 as the cun'ent sta- 
tus data. Agent processing unit 11 investigates tiie 
response data 293 to find tiiat the driver's last twice 
response to the agent's activities of the program No. 
. 001 23 are both "C" indicating tiiat tiie driver accepts the 

55 agent's suggestion or advice (see Rg; 6A). and reads 
out tiie default values (T3=2. T4=24) from study item 
data 292. From the default values T3, T4. it is confirmed 
tiiat the tenrperature relationships of T1-T2 ^ T3 and T2 
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s T4 are satisfied (see Fig. 1 1 A). 

Agent processing unit 1 1 thus selects the communi- 
cation program of No. 00123 in reference to the program 
select table 291 (at S14). After conf Inning that this pro- 
gram Is not a questioning program that awaits the 
driver's response (No at S15), it starts executing this 
program (at S16) so that the agent's activities such as 
shown in Rg. 1 1 B will be outputted ttirough cfisplay 27 
and speaker 25. More particularly, the agent appears 
with smile on display 27 and says '*lt becomes cool. May 
I suggest introducing the open air?". 

Agent processing unit 1 1 then acquires the driver's 
. response to the agenVs activities (at SIT). Since the 
program No. 001 23 is a questioning program that awaits 
tiie driver's response (Yes at S18). it controls q^eration 
of tiie air-conditioner to follow the driver's request 
When the driver's response is "acceptance", the air- 
conditioner is turned off and the front side windows are 
half-opened(atS19). 

After executing control to follow the driver's 
response, agent processing unit 1 1 stores a result of 
study depending upon the driver's response pattern. 
Examples of the study are shown in Rg. 11C. In this 
case, when the driver neglects the agent's suggestion 
(response pattern A), application of the program Na 
00123 is suspended one time. Moreover, the tempera- 
ture default values T3 and T4 to be used in tiie next 
application of the program No. 00123 are respectively 
decreased by one. so tiiat the next T3 should be rC 
and tiie next T4 should be 23''C. When tiie driver 
refuses tiie agent's suggestion (response pattern B), 
application of this program is suspended five times, and 
the temperature default values T3 and T4 to be used in 
the next application of this program are respectively 
decreased by one. so that the next T3 should be l^'C 
and the next T4 should be 23''C. When tiie driver 
accepts the agent's suggestion, ttie current default val- 
ues T3. T4 remains unchanged, and tills program is not 
suspended but executed when selected. 

Although in the above desaiption the front side win- 
dows are half-opened in response to the driver's accept- 
ance to the agent's suggestion, it may be possible that 
agent processing unit 1 1 performs more detailed control 
for opening the windows as a result of study. More par- 
ticularly, data regarding window position and degree of 
window opening are also subjected to tiie agent's study. 
For example, tiie driver may say Too much opened". 
"Close a little bit more", "Open rear side windows too", 
etc. after half-opening the front side windows in 
response to the driver's acceptance to the agent's sug- 
gestion. Such response reflects tiie driver's favorite con- 
dition regarding the window opening and. th^efore. are 
stored in tiie r^ponse data 293. The default values for 
window opening may be. for example. 1/2 for front side 
windows. When the driver says "Open rear side win- 
dows too", tills is regarded as a kind of the cun^ent sta- 
tus data (at Si 2) so tiiat a conesponding 
communication program is executed (at S16) to open 



the rear windows, with tiie corresponding agent's activi- 
ties appearing on display 27 and outputted through 
speaker 25. 

In accordance with a modified embodiment of tiie 

5 present invention shown in Rgs. 12-16. ttie agent is 
more personified by selectively controlling tiie agent's 
voice output pattern. In this embodiment, even if tfie 
same communication program is selected, tiie agent 
talks to the driver in a different voice pattern or tone, 

10 depending on the vehicle condition, the user's favorite to 
tiie agent, ttie weather condition, etc. The agent's activ- 
ity or voice pattem may be determined by also taking his 
or her character into consideration. 

In tiiis emtxxtiment. agent data memory 29 in Rg; 1 

IS is replaced with anotiier one 29a that stores voice data 
295, agent's character select table 296. voice pattem 
select table 297 and user-related inforrnation 298. in 
addition to agent program 290, program select table 
291 . study item data 292. response data 293 and image 

20 data 294, as shown in Fig. 12. Except for difference in 
contents of agent data memories 29 and 29a, tiie 
device is almost similar to that shown in Fig. 1 and 
tiierefore no description of tiie other parts and elements 
of the device is given here to avoid tedious repetition. 

25 Agent data memory 29a in this embodiment com- 
prises one or plural memory medium that can be 
selected from floppy disk, hard disk, CD-ROM. mag- 
neto-optical disk, magnetic tape, IC card, magneto-opti- 
cal card. etc. and drive unit for the memory medium 

30 used. It may be preferable ttiat study item data 292. 
response data 293 and user-related data 298 are stored 
in removable memory medium such as floppy disk and 
IC card and otiier data are stored in a hard disk. 

Voice data 295 is to output the agent's communica- 

35 tion through speaker 25. A plurality of predetermined 
words and clauses stored in the voice data 295 may be 
combined upon demand to synthesize tiie agent's com- 
munication. The driver or user may select appearance 
and voice of the agent tiirough input device 22. The 

40 agent's voice may be selected, for example, from man's 
voice, woman's voice, child's voice, mechanical voice, 
animal voice, famous actor's or actress's voice, etc. 
Once selected appearance and voice of the agent may 
be changed at any time the user so wishes. When some 

45 voice pattem is selected, it can be outputted through 
speaker 25 after being tuned by voice control unit 14 in 
accordance with the selected voice pattern. 

Agent's character select table 296 is used to deter- 
mine ttie agent's character in accordance witti ttie user- 

50 related infbrmation 298. Tiie agent's character selected 
in reference to the table 296 is used as data for deter- 
mining the voice pattern. 

An example of tiie agent's character select table 
296 is shown in Fig. 13. As shown in ttie rightnost col- 

55 umn of the table 296 in Fig. 13. the agent's character is 
determined as one selected from the group consisting 
of "obliging", "moderate", "crybaby", "strong sense of 
justice". Iseautiful secretary", "competent manager". 
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"high-school girt", etc. When a l)eautiful seCTetary" 
agent is sheeted, her peaking tends to be placid. 
When a "high-school girf is selected, she usually 
speaks cheerfully but soon become displeased when 
the vehicle is In a traffic jam or exposed to the weather. 5 
Of course, these are only examples. 

The agent's character is selected based on the 
vehicle condition such as the vehicle type and the 
number of ignition-on operation, the user's own data 
such as the user's age. sex. hobby and character, the 10 
cunrent weather corxjition. the current time. etc. These 
items may be inputted by the user through input device 
22 and/or acquired from the user's response (irputted 
through microphone 26) to the agent's question (output- 
ted through speaker 25). Some of the items may be is 
automatically stored in the table 296 by the driver's vehi- 
cle operation and/or signals from sensors. Among these 
items thus obtained, the user's own data is also stored 
in a memory section of ttie user-related information 298 
in agent data memory 29a. 20 

The voice pattern select table 297 is to determine 
the voice output pattern for tuning of the agent's voice. 
An example of the voice pattern select table is shown in 
Fig. 14. As shown in tiie rightmost column of the table 
297 in Fig. 14, ttie agent's voice pattern is determined 2s 
as one selected from tiie group consisting of loud", 
"pladd", "cool", "respectful", "tender", "kind", "trem- 
bling", "angry", "reluctant", "cheerful", "out of humor", 
etc. The voice pattern is selected, in reference to the 
table of Fig. 14. from the type of the agent's communi- 30 
cation, the vehicle condition, the agent's character, the 
user's favorite, ttie external factors, tiie current time, etc. 

In this example, the agent's communication type on 
the abscissa of the table 297 in Rg. 14 is classified into 
at least A-F types. A means route-guidance. B warning 35 
message. C explanation, D greetings, E call-back, and 
F conversation. The agent's communication type 
depends on a specific communication program to be 
selected in reference to tiie program select table 291 of 
Fig. 2. In other words, each communication program 40 
has its data regarding the communication type, that is 
one of tiie above-described types A-F 

The vehicle condition on the abscissa of the table 
297 in Rg. 1 4 includes whether or not the vehicle is in a 
high-speed (say. more than 80km/hour) drive, whether 45 
or not the vehicle is out of the prescheduled route, 
whether or not the vehicle interior is noisy (tiian a prede- 
termined noise level), the total number of ignition-on 
operation, etc. 

The agent's character on the abscissa of the table so 
297 in Rg. 14 is determined in reference to the table of 
Fig. 13. in such a manner as described above. 

The favorite on the abscissa of the table 297 in Fig. 
14 is selected by the user's input through Input device 
22 or Inputted from the user's response to the agent's ss 
question. The driver's favorite may be classified into 
"young lady", "placid lady", "showy lady", "cheerful girl", 
etc. 
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The external factors also on the absc^ of the 
table 297 in Rg. 14 include the current weather condi- 
tion (fine, cloudy, rain, snow, etc.). whether or not the 
vehicle is in a traffic jam. whether or not the veftide is 
driving on an uphill or downhill, etc. These external fac- 
tors may be automatically inputted by detection of cur- 
rent status sensor 40. 

The current time also on the abscissa of the tat>le 
297 in Fig. 1 4 indicates the current time is maning (e.g.. 
from the sunrise to noon), daytime (e.g., from noon to 
the sunset), evening (eg., from the sunset to 23X)0) or 
midnight (e.g.. from 23:00 to the next sunrise). The cur- 
rent time classification may be determined from detec- 
tion by a timer equipped in agent processing unit 1 1 and 
a calender (tiiat also stores each day's antidpated sun- 
rise and sunset time) stored in agent data memory 29. 

When the voice output pattern is determined in ref- 
erence to the tat}le 297. the agent's communication 
(speaking) is tuned accordingly by voice control unit 14. 
How to tune the agent's speaking in accordance witii 
the determined voice output pattern is not limitative. For 
example, when tiie voice output pattern is determined 
as the loudly-speaking pattern, tiie agent's communica- 
tion is outputted through speaker 25 with an inaeased 
volume. In case of the placidly- or respectfully-speaking 
pattern, voice control unit 14 controls such that tiie 
agent speaks slowly so tiiat the driver dearly under- 
stands the agent's communication word by word. The 
respectfully-speaking pattern may be tuned for giving a 
silent interval at each punctuation mark a littie bit longer 
than by tiie placidly-speaking pattern. In case of the 
cheerfully-speaking pattern, the agent's communication 
is tuned such that the agent speaks in a cheerful and 
higher voice. The agent speaks in a manner that tiie 
ending of each word is pronounced in a lower voica 

The user-related information 298 in agent data 
memory 29a stores the vehide type of tiie user' posses- 
sion, tiie user's age. sex, character, favorite to the 
agent, etc. These user-related data is used to deter- 
mine not only the agent's character in reference to the 
table 296 of Rg. 13 but also the voice output pattern in 
reference to the table 297 of Fig. 14. The user-related 
data is stored for each driver and subject to tiie study 

Rg. 15 is a flowchart showing a main routine of the 
agent control operation in accordance witti this emtxxli- 
ment. Agent processing unit 11 first initializes the device 
after ignition sensor 401 detects that the ignition switch 
is turned on, at step S1 1 1 . This step is almost identical 
to S1 1 of the flowchart of Fig. 9, description of which is 
incorporated herein. 

Agent processing unit 1 1 then identifies the current 
driver, at step S1 1 2. More particularly, when the driver's 
voice is inputted through microphone 26, it is analyzed 
by voice control unit 14 to identify who is the cunrent 
driver. The curent driver may be identified by photo 
taken by camera 28 and then analyzed by image 
processing unit 19. The weight detected by weight sen- 
sor 41 9, the driver's seat position, the angle of the room 
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mirror may also be taken into oonsideration to identify 
the current driver In a preferred emtxxliment. S12 
includes a driverndentifying communication program 
(not shown) in which the agent appears on display 27 
and speaks through speaker 25 for confirming that the 5 
current driver is just as identified. 

Once the current driver is identified and confirmed, 
agent processing unit 1 1 obtains and confirms the cur- 
rent status, at step S1 13. This step is almost identical to 
SI 2 of the flowchart of Rg. 9, description of which is 10 
incorporated herein. 

Then, at step S1 14, agent processing unit 11 per- 
forms the agent control operation depending on the cur- 
rent status confirmed at S113, in a manner to be 
described hereinlater in reference to the flowchart of is 
Fig. 16. As a result of the operation at S1 14, the agent 
makes some judgement, communication (action and 
speaking), control, study, inspection, etc. It may be pos- 
sible that the agent do nothing when so controlled by 
the current status. 20 

Then, agent processing unit 1 1 discriminates if the 
main control routine should be terminated, at step 
S115. If the main control routine should continue, the 
procedure is returned to S113 to repeat the operation 
through S1 1 3-S1 1 5. On the contrary, when ignition sen- 2s 
sor 401 detects that the ignition switch is turned off (at 
S1 13). this means that the main control routine should 
be terminated (No at 81 15). In this case, after predeter- 
mined terminative operation such as turning-off of room 
interior lamps is made at S1 14. this routine of Fig. 15 is 30 
terminated. 

Fig. 16 is a flowchart showing the agent control 
operation that is carried out in S1 14 of the main routine 
of Fig. 15. At step SI 21. agent processing unit 11 dis- 
criminates if there is any communication program to be 35 
initiated, in reference to program select table 291. This 
step is almost identical to SI 4 of the flowchart of Fig. 9. 
description of which is incorporated herein. If there is no 
communication program to be applied (No at S121), this 
sub-routine is terminated and returned to the main rou- 40 
tineof Fig. 15. 

When there is some communication program to be 
initiated (Yes at SI 21 ), it determines the agent's charac- 
ter by applying the current status (the number of igni- 
tion-on operation, current weather condition, time, etc.) 45 
that has been obtained at S1 13 of the main routine of 
Fig. 15 and the user-related information 298 in agent 
data memory 295 to the character select table 295 (Fig. 
13). at step SI 22. 

It further determines, at step SI 23. the voice output so 
pattern of the agent's speaking from the communication 
type of the program that has been confirmed at S121. 
the agent's character that has been determined at SI 22 
and the current status (vehicle status, external factors, 
time, etc.), in reference to the voice pattern select table ss 
297 (Fig. 14). 

Thus, at step SI 24. agent processing unit 1 1 reads 
out. from the voice data memory 295. voice data neces- 
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sary to synthesize the agent's speaking contents (see 
Fig. 3) of the selected communication program and 
tunes the synthesized speaking contents in accordance 
with the determined voice output pattern. Suppose that 
the program No. 0001 1 was selected at Si 21 , in which 
the agent says "Good aftemoon. Tm Lisa. May I go with 
you ?" (see Rg. 3). And. when the "respectfully-speak- 
ing" voice output pattern was selected at Si 23. agent 
processing unit 11 controls such that the agent (Lisa) 
says the above greetings relatively Stiowty. When the 
"cheerfully-speaking'' voice output pattern was 
selected, the agent says the above greetings in a cheer- 
ful and a little bit higher tone. 

After such tuning control operation at SI 24, the 
communication program selected at 81 21 Is now initi- 
ated, at step 8125. The prescribed agent's action of 
selected communication program Is shown on display 
27. while the prescribed agent' speaking thereof is out- 
putted through speaker 25 after subjected to proper tun- 
ing at 8124. 

In accordance with this embodiment, the agent Is 
more personified to make human communication with 
the driver. Even when the same communication pro- 
gram is applied, the agent's speaking pattern changes 
in response to the current status, for example. When the 
vehicle Is driving with the windows opened, the agent 
speaks budly. When the vehicle is in a traffic jam on a 
rainy day, the agent is out of humor, whereas the agent 
speaks cheerfully during driving at high speed along a 
seaside road (that can be detected by navigation 
processing unit 10) on a fine day. 

In accordance with another embodiment of the 
present invention, the agent is personified by selectively 
controlling the agent's character depending upon the 
vehicle condition or other situation concerned. In this 
embodiment, even If the same communication program 
is selected, the agent's activity should be changed in 
response to change of situation. Moreover, the agent's 
activity is determined not only the situation but also the 
past record study. 

In this embodiment, processing unit 1 (Fig. 1) is 
connected to agent data memory 29b that stores agent 
program 290, program select table 291b, study item 
data 292. response data 293. image data 294, agent's 
character select table 296 and character-determining 
data 299. as shown in Fig. 17. Program 290 and data 
292-294 are substantially Identical to those stored in 
agent data memory 29 of the first embodiment of Fig. 1 . 
Agent's character select table 296 is substantially iden- 
tical to that stored in agent data memory 29a of the sec- 
ond embodiment of Fig, 12. Accordingly, explanation 
thereof is not repeated herein. Character-determining 
data 299 is basic data for use in determining the agent's 
character in reference to character select table 296. 
Such t>asic data includes the vehicle type, the user's 
birthday, sex, character for every driver who may use 
the present system. 

Except for difference in contents of agent data 
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memories 29 and 29a. the device is almost similar to 
that shown in Rg. 1 and therefore no description of the 
other parts and elements of the device is given here to 
avoid tedious repetition. 

Agent data memory. 29b in this embodiment com- 
prises one or plural memory medium that can be 
selected from floppy disk, hard disk. CD-ROM. mag- 
neto-optical cfisK magnetic tape. IC card, magneto-opti- 
cal card, etc. and drive unit for the memory medium 
used. It may be preferable that study item data 292. 
response data 293 and character-determining data 299 
are stored in removable memory medium such as floppy 
disk and IC card and other data are stored in a hard 
disk. 

Program select table 291b is similar to table 291 of 
Fig. 2, but further includes the agent's character on the 
abscissa for some communication programs. This 
means that, in this emt)odiment. the agent's character 
that is determined in reference to character select table 
296 comprises one of factors for determining a commu- 
nication program to be applied. 

In this embodiment, as desaibed above, the agent 
activity of a specific communication program that is 
selected by table 291 may be represented in different 
manner depending on the agent's character determined 
by table 296. Suppose that a communication program 
designating the agent's activity of "falling down" is 
selected in response to the driver's emergency braking 
operation. Where the agent's character has been deter- 
mined as like a "crytsaby". the agent falls down behind 
her buttocks while snivelling or crying. A "beautiful sec- 
retary" do the substantially falls down but remains calm 
and says 'Wiy?". An agent like a "high-school girl" 
screams in doing the action of filing down". 

The flowchart of Fig. 15 also describes the main 
routine of the agent control operation of this embodi- 
ment. Description regarding the ftowchart of Fig. 15 
should be referred to in understanding the agent control 
operation of this embodiment. In this embodiment, how- 
ever, the agent control operation that is canied out at 
S1 14 of the main routine of Fig. 15 is processed as 
shown by the flowchart of Fig. 1 8. 

Now in reference to the flowchart of Fig. 18. agent 
processing unit 1 1 determines the agent's character by 
applying tiie already obtained current status (e.g.. the 
total number of ignition-on operation, the current 
weather, time, etc.) and basic data in character-deter- 
mining data memory 299 to character select tat>le 296. 
atstepS221. 

From the obtained current status data and tiie 
agent's character determined at S221. agent process- 
ing unit 1 1 discriminates if there is any communication 
program to be initiated, in reference to program select 
table 291b. at step S222. If not (No at S222). the proce- 
dure is returned to the main routine of Fig. 15. Where 
there is a communication program to be initiated (Yes at 
S222). it confirms the program number tiiereof. It furtiier 
reads out the driver's response record stored in 



response data 293 for the said program number to 
tiiereby determine if the communication program of tiie 
selected program number should be initiated or sus- 
pended, at step S223. 

5 If the answer at S223 fs No. agent processing unit 
1 1 operates to initiate the selected communication pro- 
gram so that the predetermined agenfs activities are 
shown on display 27 and outputted through speaker 25, 
at step S224. This eraUes good communication 

10 between the driver and the agent which reflects not only 
the vehicle and driver's current status but also the result 
of study of the past status data and the driver's 
response records. When the driver makes any response 
to the agent's activities made at S224. such response 

15 data is obtained through microphone 26 or input device 
22, at step S225. 

Then, at S226, agent processing unit 11 discrimi- 
nates if the communication program made at S224 
involves any suggestion, proposal or question that 

20 awaits the driver's response or answer. For example, 
agent says "Do you want to hear any radio program ?" 
or "Do you feel hungry? May I find any restraint near 
here?". The latter program may be initiated in response 
to detection of a hungry sound through microphone 26. 

25 When tiie selected communication program is not a 
questioning program (No at 8226). tiie procedure is 
advanced to step S228. When it is a questioning pro- 
gram (Yes at S226), at a step S227, agent processing 
unit 1 1 controls tiie agent's activities in accordance witii 

30 the driver's response already obtained at 8227. 
Regarding the example of operation at 8227, the 
description of 81 9 shouM be referred to. 

Agent processing unit 1 1 accumulates data regard- 
ing the execution of the selected communication pro- 

35 gram to obtain the study data, at step 8228. In 
connection with S228, the description of 820 should be 
referred to. Furtiier. in this embodiment, when tiie 
response at S225 includes any data that influences 
determination of the agent's character, it is stored in 

40 character-determining data section 299 in agent data 
memory 29. After executing the operation at 8228, tiie 
procedure is returned to the main routine of Fig. 15. 

Rgs. 19A-19C show an example of tiie agent activ- 
ities that is controlled by agent processing unit 1 1 after 

45 the ignition switch is turned on. In this example, ignition 
sensor 401 detects tiiat tiie ignition switch is turned on. 
current position sensor 21 detects the current vehicle 
position is other than the driver's home and rest posi- 
tions. These data are supplied to agent processing unit 

50 1 1 as the current status data at 81 1 3 of the main routine 
of Fig. 15. Regarding study item data 292 and respor^e 
data 293. it has confirmed tiiat this is tiie second igni- 
tion-on operation today, the last agent control operation 
was terminated at 10:05. on July 12, 1997. and the total 

55 number of ignltion-on operation amounts to 30 times. 

From these data, agent processing unit 1 1 deter- 
mines the communication program No. 00004 should be 
executed for "beautiful secretary" and No. OOOxx for 
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'*high-school girl'*, in reference to the program select 
table 291b. at S222 of the flowchart of Rg. 18. The 
agent activity to be outputted through display 27 and 
voice output device 25 is an "obeisance" that is a 
respectful bow, with saying "Hello. Tm Lisa. I will go with 
you again" in the communication program No. 0004 and 
"Hi, rm Lisa. Let's go with me" In the communication 
program No. OOOxx. After confimiing thai the selected 
communication program should not be suspended (No 
at S223), agent processing unit 1 1 starts this communi- 
cation program (at S224). 3& shown in Rg. 19B. The 
agent's obeisance is represented by a series of station- 
ary images or animation. 

Since the selected program of No. 00004 or No. 
OOOxx is not a questioning program that awaits ttie 
driver's response (No at S226), tiie procedure is 
advanced to S228 where the study data is stored and 
renewed as a result of execution of tiie selected pro- 
gram. In this example, the number of today's ignition-on 
operation is changed from tiie pres^ious value "2" to "3^ 
and the total number of ignition-on operation is changed 
from the previous value "30" to "31". Then, the proce- 
dure is returned to tiie main routine of Rg. 15. 

Fig. 20A represents a example of program select 
table 291b. and Fig. 20B represents tiie agent's activi- 
ties to be outputted in the communication program 
selected by table 291b of Rg. 20A. The communication 
program Nos. 02000-02003 shown in Fig. 20A is 
selected when the driver changes the shift lever from 
the parking position (P) to tiie driving position (D) while 
the side brake is still in operation and then pushes the 
accelerator pedal to a certain degree. The accelerator 
pedal operation, the side brake operation and tiie shift 
lever position are detected by sensors 403, 405 and 406 
(Rg. 8) in cunrent status sensor 40 (Rg. 1). which agent 
processing unit 1 1 conf irms at S1 1 3 of the main routine 
of Fig. 15. In the communication program 02000-02003. 
the agent calls the driver's attention to the current side 
brake position and immediate release thereof, which is 
accomplished by the agenfs activities shown in Rg. 
20B. 

When the "obliging beautiful secretary" has been 
selected as tiie agent's character (at S221 of Rg. 18), 
the communication program Na 02000 is selected in 
reference to program select table 291b of Rg. 20A (at 
S222). As shown in Fig. 208, this communication pro- 
gram designates tiie agent activity such that tiie agent 
warns by saying TTie side brake is still in operation. 
Release it, please", in a persuasive tone. Also, the 
agent's activity is shown on display 27 just like such tiiat 
her right hand is pulled back by tiie side brake. 

When the "high-schod girt' has been selected, the 
communication program No. 02001 is applicable. In this 
case, tiie high^ool girl on display takes tiie same 
reaction while saying "Oh nol The side brake is still 
working. Release it please" in a cheerful tone. When tiie 
"crybaby" has been selected, the communication pro- 
gram No. 02002 is applicable so that the crybaby takes 



the same reaction witii a tearful face and says "CXichl 
The side brake is still in operation. Release it. please" in 
a sadly and entreating tona When the "iplacid" agent 
has been selected, the commurtication program No. 

5 02003 is applicable. In this case, the agent takes the 
same reaction and says "May I suggest you releasing 
the side brake" in a placid tone. 

In the foregoing embodiment the agenfs character 
changes depending upon the current status, the user- 

10 related data, the weather condition, time, etc. However, 
in a more simple manner, the agent's character may be 
selected from predetermined several characters 
depending on the driver's familiarity to the agent sys- 
tem. The driver's familiarity to the agent system is calcu- 

is lated in a manner to be desaibed hereinlater. The agent 
is "placid" to tiie driver who is much familiar to tiie agent 
system, whereas an "obliging" agent appears to the 
unfamiliar driver. The obliging agent provides sugges- 
tion, advice, help, explanation, etc. in a meddlesome 

20 manner rather than by tiie placid agent. 

In this modified embodiment, agent data memory 
29c stores familiarity data 300, as shown in Rg. 21. 
Agent's character select table 296 and character-deter- 
mining data 299 in agent data memory 29b are not 

2s present. Familiarity data 300 is stored for every driver 
who may use tiie agent system, and renewable by tiie 
past record study In this modified embodiment, tiie 
agent character is "obliging" or "placid" so that only 
tiiese two characters are included in tiie column of 

30 "Agent's Character" in program select table 291b (Rg. 
20A). 

Rg. 22 shows an example of familiarity data 300. 
As shown, familiarity data 300 desaibes tiie program 
number, tiie flag, the last five time response period, the 

35 average response period, tiie shortest response period 
and the number of response cancellation for each com- 
munication program. The response period means a 
period of time from when the agent makes some ques- 
tion until when the driver makes some response thereto, 

40 or a period of time when the driver is expected to make 
some operation until the said operation is actually 
taken. The response periods for the latest five time 
responses are recorded in familiarity data 300. The 
average response period is tiie average of the five 

45 records. The shortest response period is tiie shortest 
one of the five records. When tiie driver cancels his or 
her response or operation while the selected communi- 
cation program is still working, it is accumulatively 
recorded as the number of response cancellation. The 

so flag describes how many times tills communication pro- 
gram has been selected. Familiarity data 300 is pre- 
pared for every communication program tiiat requests 
tfie user's response, fbr every user of tiiis agent system. 
The familiarity level is calculated from tiiese data 

55 and also described in familiarity data 300. Of course, 
the familiarity level is zero when the flag is zero. Each 
time the communication program is selected, tiie famili- 
arity level is inaeased by 1 point The average response 
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period becomes shorter than a predetermined period, 
the familiarity level is increased by 2 point. The shortest 
response period becomes lower than a predetermined 
period, it is inaeased by 1.5 point. 

On the other hand, when the average response 
period becomes longer than a predetermined period, 
the familiarity level is decreased by 2 point The shortest 
response period exceeds a predetermined period, it is 
decreased by 1 .5 point. It is deaeased by 1 point, when 
the response cancellation is made by the driver. The 
respective points to be Increased or decreased may be 
determined as desired. 

In this modified embodiment, agent processing unit 
11 determines the agent's character by confirming the 
familiarity level of the selected responsive program and 
of the current user, at S221 of Rg. 1 8. By way of exam- 
ple, when the familiarity level in familiarity data 300 (Fig. 
22) is below 40. the agent' character Is determined as 
"obliging", whereas it is 70 or more the agent's charac- 
ter is determined as "placid". In accordance with the 
agent's character thus determined at S221, a specific 
communication program is selected at S222 followed by 
the succeeding steps. 

At S225. agent processing unit 11 confirms the 
driver's response to the agent's activity in the running 
program, which is made through miaophone 26 or input 
device 22. It also confirms the response period from 
when the agent's activity is outputted through display 27 
and speaker 25 until when the driver's response thereto 
is obtained. From these, agent processing unit 11 
rewrite or renew the fomillarity data 300. More particu- 
larly, in the column of the last five time response period, 
the oldest record is annulled, the remaining four records 
are moved to the older boxes one by one and the 
obtained response period is recorded as the latest one. 
Then, the average response period and the shortest 
response period are calculated and recorded. From the 
renewed records, the familiarity level is calculated by a 
predetermined formula and the newly obtained familiar- 
ity level is recorded in the familiarity data section 300 
which should be used in the next determination of the 
agent's character. 

In accordance with still another embodiment of the 
present invention, a plurality of agents may appear on 
display 27 and/or oiitputted through speaker 25 so that 
the user can easily understand the communication, 
message, suggestion, etc. from the agents. 

In this embodiment, processing unit 1 (Fig. 1) is 
connected to agent data memory 29d that stores agent 
program 290. program select table 291 . study item data 
292, response data 293. image data 294, help-agent 
program 301. help-agent program table 302 and famili- 
arity data 300. as shown in Fig. 23. Program 290. table 
291 and data 292-294 are substantially identical to 
those stored in agent data memory 29 of the first 
embodiment of Fig. 1. Familiarity data is substantially 
identical to that stored in agent data memory 29c of the 
preceding embodiment of Fig. 21 and an example 



thereof is shown in Rg. 22. Accordingly, explanation 
thereof is not repeated herein. 

Help-agent program 301 comprises help-agent 
processing program and help-communication program 

5 especially for each responsive communication program 
stored in agent program section 290. Help-agent 
processing program Is used to carry out help-agent 
operation which provides explanation a other suitable 
communication for easier understanding of the main- 

10 agenf s activity, while some responsive communication 
program is working. Help-agent communication pro- 
gram describes the help-agent*s activities to be repre- 
sented through display 27 and speaker 25. 

Help-agent program 301 stores plural kinds of voice 

15 data fbr each main agent's voice of the main-communi- 
cation program, one of which may be selected by the 
driver's operation through input device 22. fbr example, 
so that the driver distinguishes the main-agent's and 
help-agent's voices both outputted through speaker 25. 

20 For example, the main-aigent is female, the help-agent 
shoukl be male. In a prefen-ed example, after the main- 
agenf s voice is determined, agent processing unit 1 
represents on display 27 a list of voice patterns for the 
help-agent which is distinguishable from the main- 

25 agent's voice, and the driver select one through input 
device 22. In the absence of the driver's instruction, 
agent processing unit 1 automatically determines the 
help-agent's voice that is distinguishable from the main- 
agent's voice. Likewise, the help-agent's appearance 

30 should be well distinguishable from the main-agent's 
appearance. 

Help-agent program table 302 represents cone- 
spondence between the (main-) agent program number 
and the help-agent program number, and the familiarity 
35 level tiiat Is critical to carry out the help-agent program. 
An example of the help-agent program table 302 is 
shown in Fig. 24. and some of the help-agent's activities 
in tiie help-agent programs in Rg. 24 are represented in 
Fig. 25. 

40 For example, when the agent's communication pro- 
gram No. 00123 (that suggests stop of the air-condi- 
tioner) is in operation, tiie help-agent program No. 
50010 is applicable provided tiiat the driver's familiarity 
to this communication program is 50 or lower. In tills 

45 help-agent program, the help-agent points his or her 
forefinger at the main agent (Lisa) on display 27 and 
says "Lisa stops tiie air-conditioner and opens the front 
windows half through speaker 25. Even when tiie same 
communication program No. 00123 is working, when 

so tiie driver's ^miliarity is higher than 50, the conespond- 
ing help-agent program No. 50010 is not operative and 
no help-agent appears. 

Rg. 26 Is a flowchart of the main routine of tiie 
agent control operation in accordance witii this embodi- 

55 ment. Steps S301-306 of this routine are substantially 
identical to S1 1-16 of the flowchart of Fig. 9, so that 
description of the latter should be referred to as desaip- 
tionofthe former. 
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After initiating ttie applicable comniunication pro- 
gram at S306. agent processing unit 1 1 checks and 
conf irnis the familiarity data 300 of the said communica- 
tion program in agent data memory 29. at step S307. At 
the same time, the familiarity flag is set to 1 . Then, the s 
program nun^er of the selected communication pro- 
gram and the confirmed ^nrdliarity data are applied to 
the he^agent program table 302 (Fig. 24) to discrimi- 
nate if there is any help-agent program to be initiated, at 
S308. If not (No at S308), the procedure is advanced to 70 
step S310 where the driver's response is obtained. 
Steps 310-316 are substantially identical to S17-23 of 
the flowchart of Fig. 9 and, ttierefore. no detailed 
description is given herein. 

When some help-agent program to be initiated in is 
the table 302 (Yes at S308). tiiis is executed so that the 
prescnlied help-agent's activity (action and utterance 
such as shown in Fig. 25) is outputted through display 
27 and speaker 25. at step S309. 

In accordance witii this embodiment, the help- 
agent appears and supports the nnain-agent*s activity 
when the driver is not so much familiar to the present 
agent system, whereby the driver can make more 
smootii communication with the agent and more suita- 
ble response to the agent's question or suggestion. 

Figs. 27A-27C illustrate an example of tiie agent 
control operation made by the flowchart of Fig. 26. This 
example is executed when the air-conditioner sensor 
418 detects "ON", the room temperature sensor 413 
detects tiie room temperature T1 and the open air tem- 
perature sensor 414 detects the open air temperature 
T2. as ttie current status obtained at S302. Operation at 
S303 reveals tfiat the last two response to the commu- 
nication program No. 00123 are both C indicating the 
driver's acceptance so tiiat tiie default temperature val- 
ues T3=:2 and T4=24 have been read out from tiie study 
item data 292. Consequently. The relationship of (T1- 
T2)^ T3 and T2 ^ T4 are both satisfied, as shown in Fig. 
27A. 

From tiiese operation at S302 and S303. agent 
processing unit 11 finally selects tiie communication 
program No. 00123 at S304. TTien. after confirming ttiat 
this program is not subjected to suspension at S305, it 
starts the communication program No. 00123 for repre- 
sentation of the (main-) agent's activity through display 
27 and speaker 25. at S306. 

Next, agent processing unit 1 1 confirms the driver's 
familiarity to the selected communication program at 
S307 and finds out an applicable help-agent program at 
S308. Since tiie driver's familiarity to the communication 
program No. 00123 is 25 (Fig. 27A) in tiiis example, and 
the communication program No. 00123 has a corre- 
sponding helpnagent program No. 50010 in the help- 
agent program table 302 (Fig. 24), tiiis help-agent pro- 
gram No. 50010 is executed at S309. Accordingly, the 
help-agent H appears on display 27 in a predetermined 
appearance and speaks tiirough speaker 25 in a prede- 
termined manner and tone. In tills example, tiie main- 
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agent (Lisa) E says "It t>ecomes cod. I may suggest 
introducing the open air." and then the help-agent H 
says "Lisa stops the air-conditioner and opens the front 
windows haT. The help-agenfs activity provides sup- 
port and explanation of tiie main-agent's activity to the 
driver. Data necessary to display tiie help-agent activity 
is included in tiie image data 294. . 

At S3 10 agent processing unit 11 obtains the 
driver's response. Since the communication program 
No. 00123 is a questioning or responsive program (Yes 
at S311). it confrols such tiiat the air-conditioner is 
stopped and tiie front windows are opened half in 
response to the driver's acceptance to the agenfs sug- 
gestion (Rg. 27B).atS312. 

The driver's response is stored in response data 
293 for the communication program No. 00123. at 
S313. Since response data 293 stores the latest two 
response, the older response data is deleted and the 
newly obtained response data is stored together with 
one remaining data. As shown in Rg. 270. when the 
driver's response is "neglect", application of this com- 
munication program is suspended one time, and the 
default temperature values T3 and T4 to be next applied 
are decreased by one. respectively When the driver 
25 shows "refusal" application of tiiis communication pro- 
gram is suspended five times, and the defoult tempera- 
ture values T3 and T4 to be next applied are decreased 
by one, respectively In case of "acceptance", tiie same 
default values remain unchanged in the next applica- 
30 tipn. 

Operation at S3 12 also includes re-calculation of 
the driver's familiarity to this communication program 
No. 00123. Mae particularly, tiie familiarity data 300 
(Fig. 22) is rewritten such that tiie oldest response 
35 period is deleted and the newly ob^ined response 
period (i.e., a period of time from when tiie agent 
appears on display 27 and until any driver's response is 
obtained). From the rewritten five response period data, 
new average response period, new shortest response 
40 period and new familiarity level are calculated and 
recorded in the respective memory areas. 

When the driver's familiarity to tiie communication 
program No. 00123 is grown to above 50, agent 
processing unit 11 discriminates there is no applicable 
45 help-agent program at S308, because the help-agent 
program table 302 teaches tiiat the help-agent program 
. No. 50010 corresponds to the communication program 
No. 00123 but should be executed only when tiie 
driver's familiarity is lower tiian 50. In this case, only tiie 
so main-agent (Lisa) appears on display 27 while saying 
tiie prescribed suggestion to introduction of the open 
air, in accordance witii tiie communication program No. 
00123. Then, agent processing unit 11 obtains tiie 
driver's response at S310, make suitable control in con- 
55 fbrmity to the driver's response at S31 1 and rewrites tiie 
response data 293 and the familiarity data 300, as in the 
above-described manner. 

The above-described embodiment may be modified 
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as follows. In th^ modif k^atioa a plurality of sub-agents 
may appear depending on the driver's trend. A typical 
example of the sub-agent is a short-cut agent who 
appears, when the driver is much familiar to a ispecific 
communication program, for short-cut or skip of a part 5 
of the agent's lengthy activity of the same communica- 
tion program. 

In this nrodification, agent data memory 29d of Fig. 
23 is replaced with agent data memory 29e of Rg. 28. 
In agent data memory 29e are stored agent program 10 
290. program select table 291. study item data 292. 
response data 293, image data 294, sub-agent program 
303, sub-agent program table 304. trend data 309 and 
frequent program data 305, as shown in Fig. 28. Since 
the former five memory corrtents have been described is 
in connection with the preceding embodiments, the last 
two will be described in detail. 

Sub-agent program 303 comprises sub-agent 
processing program and sub-agent communication pro- 
gram especially for each communication program 20 
stored in agent program section 290. Sub-agent 
processing program is used to carry out sub-agent 
operation wherein a sub-agent appears together with a 
main agent depending on the driver's trend during oper- 
ation of some communication program. Sub-agent com- 25 
munication program describes the sub-agent's activities 
to be represented through display 27 and speaker 25. 
Sub-agent processing program at least includes a pro- 
gram for achieving the short-cut agent operation that 
short-cuts or skips the main-agent's lengthy activity so 
when the driver is so familiar thereto. 

In another example, a specific trend value is 
obtained from the driver's birthday and the traffic jam 
condition so that the today's fortune is noticed to the 
driver when the vehicle is in a heavy traffic jam. ss 

In still another example, data regarding the driver's 
favorite professional baseball team and the radio pro- 
gram broadcasting the said team's games are stored as 
the driver's trend data 309. The sub-agent program is 
executed at the time when the radio program is to start ^ 
so that the sub-agent announces the radio program and 
the opponent team and ask if the radio receiver should 
be turned on. The radio program data may be inputted 
by tiie driver or automatically received from communica- 
tion control unit 24. 45 

Another driver's trend data may be determined from 
the current vehicle position and tiie places of the driver's 
interest so tiiat tiie sub-agent draws tiie driver's atten- 
tion to some recorded place near the cunrent vehicle 
position. Still another driver's trend data may be deter- so 
mined from the driver's driving career and tiie current 
vehicle position so tiiat the sub-agent announces "Be 
careful, you are driving on a winding road", for example 

Most typical example of tiie sub-agent program will 
be tiie short-cut agent program to be described In nrK>re ss 
detail. 

After executing tiie sub-agent program at S408, 
agent processing unit acquires the driver's response, at 




S409. As described before, if tiie driver's trend data is 
lower than tiie predetemiined value (No at 8407), no 
suthagent program is executed arxi the driver's 
response is obtained at S409. The following steps 
S410-S415 are substantially identical to S18-S23 of Rg. 
9. 

The activity In the sub-ag^ program will now be 
described in detail. In tiie following desaiption. the sub- 
agent is the short-cut agent As shown in Rg. 33, tiie 
communication program for menu sdection comprises 
multi-stage operation. The first-stage operation is mode 
select operation. The second-stage operation Includes 
goal select mode and system select mode, one of which 
may t>e selected by the driver's operation through input 
device 22 when the first stage operation is working. 
When the goal select mode is selected, tiien the driver 
may select and enter one of tiiird-stage operation, that 
is. how to select tiie goal, from its category (highway 
tamps, railroad stations, hotels, restraints, golf courses, 
etc.). from its phone number or from its spelling. When 
tiie system select mode is selected, then the driver may 
select one of tiiird-stage operation for setting display, 
voice or communication in the agent system. In the nor- 
mal operation, operation begins with at the first stage, 
proceeds through the second stage and terminates at 
the third stage. 

Rg. 34 is similar to Rg. 33 but different in that tiiere 
Is a short-cut program shown by dotted lines. In tiiis 
embodiment, after selecting the first-stage operation for 
mode selection, tiie driver may skip over tiie second- 
stage operation to select one of tiie third-stage opera- 
tion by using the short-cut program. The short-cut pro- 
gram is applicable when the driver is so familiar to tiie 
communication program, or when the driver's familiarity 
to the communication program is 75 or more in this 
example. 

When the driver so operates to start tiie navigation 
system, agent processing unit 1 1 first perfornns the first- 
stage operation, tiiat is tiie mode select operation. The 
communication program No. 00540 now becomes oper- 
ative (at S406) so tiiat the main agent appears on dis- 
play 27 and says "May t soon start navigation program 
or provide you system select program?". Since tiiis 
communication program has no corresponding short- 
cut program (No at S407). it awaits tiie driver's 
response at S409. When the driver requests navigation 
program In reply to tiie agent's activity of ttiis program 
No. 00540 through input device 22 or microphone 26, 
tills is confirmed at S409. This program No. 00540 is not 
subjected to the short-cut control (No at S410). and 
agent processing unit 1 1 carries out tiie following oper- 
ation at S411-415, tiien the procedure is returned to 
S402. 

The communication program No. 00550 for select- 
ing the goal input mode is now executed at S406. This 
means that tiie operation proceeds to tiie second-stage 
operation of Fig. 33. In this program No. 00550. the 
main agent appears on display 27 and says "Select your 
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goal input mode. From its category, from its phone 
number or from its spelling?*, as shown in Rg. 36. 
Agent processing unit 11 then conpares the driver's . 
familiarity (75) with the predetermined critical value (70 
in Rg. 29) at S407. Since the former is larger than the 5 
latter in this exanrrple (Yes at S407). in reference to the 
short-cut program table 304 in Rg. 29, agent processing 
unit 11 executes the short-cut operation in accordance 
with one of the short-cut programs selected in accord- 
ance with the frequent program of the last five program 10 
selection (at S408). 

Rg. 35 is a flowchart of the short-cut agent control 
operation at S408. As shown, agent processing unit 1 1 
obtains which is the most frequent program, at 8461 
from the frequent program data 305 in agent data mem- is 
ory 29. Since the most frequent program is No. 00580 in 
the exanple of Rg. 31 » which means that the driver nor- 
mally inputs the goal from its category in response to 
the agent's activity of the mode select program No. 
00540. agent processing unit obtains the short-cut pro- 20 
gram No. 80010 in reference to Rg. 30. at S482, and 
then executed the same at S483. By executing the 
short-cut program No. 80010. the short-cut agent S 
appears on display 27 beside and behind the main 
agent E. and says 'you can readily enter the categorical 2s 
input mode.** following the main agent's speaking, as 
shown in Fig. 36. The short-cut agent then awaits the 
driver's response. 

When the driver responsively says "Restraints'*, this 
is confirmed by the short-cut agent. In this case, agent 30 
processing unit 11 compulsory terminates the main 
agent communication program No. 00550 without reev- 
ing any response from the driver, and execute another 
communication program No. 00600 (at S404). The com- 
munication program No. 00660 is prepared as the 35 
fourth-stage operation after selecting the categorical 
input mode operation 00580 as one of the third-stage 
operation (Figs. 33 and 34). Thereafter, the current 
vehicle position is detected and the restraint map near 
the detected cun-ent vehicle position is shown on dis- 40 
play 27 for selective input by the driver. 

The driver may say "Category" in response to the 
main- and sub-agenfs activities at S483. In this case, 
the procedure is advanced to the categorical input 
mode operation of the program No. 00580 which is exe- 4s 
cuted at S404. The list of selectable categories is repre- 
sented on display 27 and the main agent says "Select 
category of the goal". When the driver selects a specific 
input mode, the operation proceeds in tiie above- 
described manner. so 

On the other hand, if tiie agent's familiarity is 
smaller than 70 (No at S407). agent processing unit 1 1 
does not execute tiie short-cut program. In this case, 
only the main agent appears and says in accordance 
witii the main agent communication program that has ss 
been determined at 8404, as shown in Fig. 37. 

In this embodiment, the short-cut agent appears 
together with the main agent to suggest the short-cut 



operation to the driver familiar to tine agent system. 
When the driver forgets the short-cut operation, he or 
she may follow the main agent's activity to successively 
carry out the multi-stage operation stage by stage. 

The sub-agent may act independently of the main 
agent's activity For example, the main agent acts as a 
guide or navigator whereas the sub-agent notices the 
current vehicle condition to the driver, and the vice 
versa 

In accordance with yet another embodiment of the 
present invention, one or more of personified agent 
appears on display 27 together witii a predetemnined 
background so that the user can easily understand the 
communication, message, suggestion, etc. from the 
agents.. The agent's activity and tiie background are 
determined depending on the current status of tiie vehi- 
cle, the driver and passengers, etc. as well as the 
driver's response. 

In this embodiment, processing unit 1 (Fig. 1) is 
connected to agent data memory 29f that stores agent 
program 290. program select table 291 , study item data 
292f, response data 293. image data 294, background 
select table 306 and response recognition data 307. as 
shown in Rg. 38. Program 290. table 291 and data 293- 
294 are substantially identical to tiiose stored in agent 
data memory 29 of the first embodiment of Fig. 1. 
Accordingly, explanation ttiereof is not repeated herein. 

The study item data 292f in agent data memory 29f 
is shown by example, in Rg. 5 which is siightiy different 
from the study item data 292 (Fig. 5) in agent item data 
292 (Rg. 1). That is. the study item data 292f also 
includes audio instruments equipped in the vehicle and 
their operating condition. Audio instruments may com- 
prise a radio receiver, a CD player, a MD player, a cas- 
sette tape player, a TV set etc. The audio instrument 
operating condition describes when and where any 
audio instrument is turned on. If the radio or TV is 
turned on, the selected channel is also recorded. The 
audio instrument operating condition have five time 
records. TTie remaining items in the study item data 
292f have been described in connection witti the first 
embodiment. 

In this embodiment, image data 294 include data 
for representing tiie background on display 27. as well 
as data for the agenfs appearance and activity The 
photo image taken by camera 28 is analyzed by Image 
processing unit 13 to be used to represent the agent's 
activity or the background. The background Image data 
294 may be photo, animated or any other image data 
representing sunrise, sunset, starlight night, snowy 
mountain, autumnal tints, goH course, plateau, etc. 
Among tiiese. sunrise, sunset and starlight night back- 
grounds may represent the cun-ent time, snowy moun- 
tain and autumnal tints may represent tiie cun'ent 
season and golf course and plateau background may 
represent the goal inputted by the driver in the naviga- 
tion system. The background may also be tiie agent's 
belongings, such as nameboards showing tiie driver's 
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selectable response or answer 'VES" arxi "NO", which 
is appScable when some questioning communication 
program Is in operation. When the driver is Gstening to 
the music through CD or radio, a t>ackground like a 
musical note may be shown on display 27. 

An example of the background select table 306 Is 
shown in Rg. 40. The items on the ordinate are various 
background, among which one or more of specific back- 
ground is selected by the items on the abscissa. The 
items on the abscissa include, in this exarrqple. the cur- 
rent time, the current season, the Instruments now In 
operation, the voice recognition status, the most fre- 
quent recognized word, the goal, etc. These items may 
be detected by various sensors included in status sen- 
sor 40. alone or in conri)ination with the contents in the 
study item data 292 and/or response data 293. In this 
table, the Item symtx>lized by "O" is a necessary con- 
dition for selecting the background on the ordinate. The 
item with no symbol should not be satisfied to select the 
background on the ordinate. 

Figs. 41 A-41 C show examples of the background to 
be selected by the bad^round select table 306, which 
is represented together with the agenfs activity on dis- 
play 27. In Fig. 41 A, the background with a red frame 
(see Fig. 40) is selected when the driver's voice Is not 
recognizable by the agent The agent In Fig. 41 A is sit- 
ting down, which represents that the vehicle Is now at a 
standstill. The background is shown in Fig. 41 B with a 
yellow frame (see Fig. 40) which indicates that the 
agent awaits the driver's answer by saying "YES" or 
"NO". In this background condition, the driver's voice 
other than "YES" and "NO" can not be recognized by 
the agent. The agent in Fig. 41 B carries two name- 
boards of "YES" and "NO" while saying "Please answer 
by saying "YES" and "NO" to my question", for example. 
This agent's activity is outputted when some question- 
ing communication program that request the driver's 
answer. The agent's belongings such as the above 
nameboards should be a kind of the background, as 
noted from Fig. 40. 

When the communication program for goal input is 
running and the driver selects to input the goal by its 
category and then by its distinct, the agent appears on 
display 27 witii a nameboard of "DISTRICT" for urging 
the driver to input a specific name of the distinct of the 
goal, as shown In Rg. 41 C. The driver's input may be by 
voice through microphone 26 or by operation of input 
device 22. At the same time, a pertinent map is shown 
on display 27. and the frame of display 27 Is colored 
green for indicating that any driver's voice Is recogniza- 
ble. In this example, ttie "DISTINCT" nameboard, tiie 
background map and tiie green frame are represented 
as the backgrounds that have been selected by the 
table 306 (Fig. 40). Fig. 8D merely shows another 
example of the background with a green frame indicat- 
ing voice recognizable condition, as above described. 
The agent Is standing upright in Figs. 41B-41C which 
Indicates tiiat the vehicle Is running. 



The response recognition data 307 represents a 
result of the agenf study of the response from the 
driver. The response recognition data 307 is prepared 
for each driver and rewritten as the driver makes some 

5 fresh response. 

An example of the response recognition data 307 is 
shown in Fig. 42. As shown, the response recognition 
data 370 Includes the response recognition results, dis- 
tribution of the recognized response patterns and the 

10 most recognizable response pattern, respectively for 
affirmative and negative responses. The affirmative 
response may be "Yes", "OK". "Well". "Certainly". 
"Sure", etc. and the negative response may be "No", 
"You're Kidding", "Not at all", etc. The response recogni- 

15 tion results records in respective answer columns "O " 
or "X" symbols indicating if such driver's answer could 
be con^ectiy recognized in the agent system, for past 10 
times recognition. The driver's answer inputted tiirough 
microphone 26 is processed by voice processing unit 

20 14. and a signal after voice processing is supplied to 
agent processing unit 11 for executing some control 
operation. If the driver's anwer has been recognized In 
error, the driver's should make any surprising response 
to the executed control operation. From this, it could be 

25 judged tiiat the last time voice was recognized In error, 
the distribution percentage is determined by divid- 
ing the total number of voice acquisition by the number 
of connect voice recognition, followed by being multiplied 
by 1 00. The driver's answer having the greatest dista-ibu- 

30 tion percentage is recorded as the most recognizable 
answer. Thus, two nameboards describing the most rec- 
ognizable affirmative and negative answers in tiieir 
response recognition data 307 (Fig. 42) are shown as 
the background of display 27. In case of Fig. 41 B, for 

35 example. 

The main routine of the agent operation of this 
embodiment is the same as shown in Fig. 1 5 and, tiiere- 
fore. description regarding the ftawchart of Fig. 15 
should be referred to in understanding the agent control 

40 operation of this embodiment. In this embodiment, how- 
ever, tiie agent cont'd operation that is carried out at 
S1 14 of the main routine of Fig. 15 is processed as 
shown by the flowchart of Fig. 43. In he flowchart of Rg. 
43. steps S501 -502 are substantially identical to SI 4-1 5 

45 of the flowchart of Rg. 9. so ttiat description of the latter 
should fc>e referred to as description of the former. 

After confirming that the applicable communication 
program should be executed in this operation (No at 
S502), agent processing unit 11 discriminates if ttie 

so applicable communication program Is an instrument 
control program, at step S503. The instrument control 
program means a program to be executed to control 
operation of the vehicle or any instrument mounted to or 
equipped in tiie vehicle. If this is the case (Yes at S503). 

55 tiie said program Is executed to cany out necessary 
operation under control, at S509. The result of the con- 
trol operation is stored In the study item data 292. 
When tiie applicable communication program Is not 
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an instrument control program (No at 8503). at step 

5504, agent processing unit determines a display back- 
ground, in reference to the bad^round select table 306 
(Fig. 40). from the cunrent status (the current time, sea- 
son, instruments In operation, status of driver's voice 5 
recognition, goal irputted in the navigation system, etc.) 
and the response recognition data 307 in agent data 
memory 29. The applicable communication is executed 

by displaying the selected background, together with 
the prescribed agent's activity, on display 27. at S505. w 
The agent may also talk to the driver through speaker 
25 in accordance with the contents of the applicable 
communication program. .Thus, the agent^s activity 
(action and voice) such as shown in Rgs. 4^-410 is 
shown on display 27 with a suitable background that is is 
pertinent to the agent's activity. 

When this communication program is not a ques- 
tioning or responsive one requiring the driver's any 
response (No at S506), the procedure is returned to the 
main routine. When this is the case (Yes at S506), agent 20 
processing unit 11 acquires the driver's response or 
answer through microphone 26 and voice processing 
unit 14 or through the driver's operation of input device 
22. at step S507. The acquired response is incorpo- 
rated into response data 293 at step S508, then the pro- 2s 
cedure is returned to the main routine. The data 
accumulation or the past record study at S508 has been 
described in connection with the preceding embodi- 
ments. In this embodiment, when the driver's response 
acquired at S508 is any reaction to the agent's activity 30 
that is executed in response to the last time driver's 
affirmative or negative answer in reply to the agent's 
questioning activity, this reaction is stored in the 
response recognition data 298 so that the distribution 
and the most recognizable answer are rewritten when 3s 
necessary, as having been described. 

Figs. 44A-44C show examples of the agent's activ- 
ities shown on display 27. with the pertinent back- 
ground, in the convnunication program executed at 

5505. Rg. 44A represents the cun-ent status data 40 
acquired at 81 13 of the main routine of Rg. 15 and con- 
tents of the study item data 292 and the response data 
293 in agent data memory 29. The acquired current sta- 
tus data includes the current time (17:00), the cun'ent 
vehicle position (Tokyo) that is detected by the vehicle 4S 
position sensor 21 . The study item data 292 and the 
response data 293 storing the past record study repre- 
sents in tiiis example that the driver has most frequently 
been listening to the FM radio program "J-WAVE" dur- 
ing 17:00-1 8m so 

From these data in Fig. 44A. agent processing unit 
1 1 selects a communication program in reference to the 
program select table 291 at 8501 . In this case, the conv 
munication program No. 00501 in which the agent sug- 
gests the driver listening the radio program is selected ss 
at 8501 . This program is not subjected to suspension or 
skip, which is confirmed at 8502. Then, it is confirmed 
that this program is not an instrument control program at 
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8503. Agent processing unit 11 then detennines a 
t)ackground to be displayed together with the agent's 
activity in the selected communication program No. 
00501, at S504. The background selection is made in 
reference to the table 306 of Rg. 40. For instance, since 
the cunrent time is between 1 5:00-18:00, a sunset back- 
ground is selected. Further, tiiis program No. 00501 
involves tiie agent's speaking, and the driver's voice 
spoken at the same with the agenfs speaking woukJ 
hardly be recognized. Accordingly, when executing the 
program involving the agent's speaking, display 27 has 
a red frame indicating the voice unrecognizable situa- 
tion now. Thus, a red frame k}ackground is also selected 
atS504. 

After one or more background has been selected in 
this way, the communication program No. 00501 is exe- 
cuted at 8505 so that the agent's activity and the 
selected background are shown on display 27, as 
shown in Fig. 44B. More particularly, the agent appears 
on display 27 with a sunset background and says Tou 
may listen to the radio program?". This communication 
program does not need the driver's response (No at 
8506) so that tiie procedure is returned to the main rou- 
tine of Fig. 15. 

In the next application of the main routine of Fig. 1 5, 
tiie driver's answer awaiting condition created by the 
program No. 00501 is newly confirmed at 81 13 of Fig. 
15 as one of the cunent status. As a consequence, 
another communication program is selected at 8501 for 
obtaining the driver's answer in response to tiie agent' 
speaking (Fig. 44B). Through 8502 (No) and 8503 (No), 
at 8504, agent processing unit 1 1 determines the back- 
ground to be applied in executing this program, in refer- 
ence to the background select table 306 (Fig. 40). In this 
case, the current time is within the period 15-18, the 
program to be execijted requires recognition of the 
driver's affirmative or negative answer, and the most 
recognizable answers (in the re^onse recognition data 
307 of Fig. 42) are Tes" and "No". Accordingly, a sunset 
background, two nameboards of TES" and "NO" as the 
agent's belongings and a yellow frame (indicating that 
only tiie driver's affirmative and negative answers can 
be recognized) are selected as backgrounds, which are 
shown on display 27, togetiier with a silent agent, as 
shown in Fig. 44C, when this program is actually exe- 
cuted at 8505. 8ince tiiis program is a questioning pro- 
gram (Yes at 8506), the driver's answer is obtained at 
8507. In tiiis example, it is supposed ttiat the driver says 
"Yes" in response to the agent's activity of Rg. 44C. This 
response data 293 is renewed accordingly at 8508, and 
the procedure is returned to the main routine of Fig. 15. 

In tiie next application, tiie fact tiiat the driver made 
an affirmative answer in response to tiie agent's activity 
in the previously executed communication program is 
confirmed at 81 13 as one of the current status. As a 
consequence, still another communication program is 
selected at 8501 for turning tiie radio receiver on and 
selecting the radio channel (Rg. 44B). This program 
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should not be suspended (No at S502). Since this is an 
instrument control program (Yes at 8503), the racfio 
receiver is controlled accordingly at S509. In execution 
of this program, the radio receiver is automatically tuned 
to the nfK)St frequently selected channel in the current 5 
area ard at the current time in accordance with the 
study Item data 292. Thereafter, data regarding the 
radio in the study item data 292f (Rg. 39) is rewritten to 
include data in the latest operation. 

Unless the driver makes any reaction to ask inter- 10 
ruption of operation of the radio receiver by saying "No** 
or "Stop", for example, agent processing unit 11 dis- 
criminates that the driver's answer ("YesT has been cor- 
rectly recogniized. Thus, the response recognition data 
307 (Rg. 42) should be renewed by incorporating this 15 
answer at S508. and the procedure is returned to the 
main routine of Rg. 15. 

In the next application, the operating condition of 
the radio is confirmed at S1 13 as one of the current sta- 
tus. Still another communication program may be 20 
applied to represent the current vehicle status (the sta- 
tus of the radio, in this case) at 8501 . Through 8502 and 
S503. one or more pertinent background is selected at 

8504. In this case, the present time is 1 7:00, the radio is 

in operation and any voice Is recognizable, so that .a 2s 
sunset background with a green frame (indicating any 
voice recognizable condition) is shown on display 27 
together with a silent agent can'ying a radio/cassette, at 

8505. Since this program is not a questioning program 
(No at 8506), the procedure is soon returned to the 30 
main routine of Rg. 15. 

In accordance with the above-described embodi- 
ment, not only the agent but also the background perti- 
nent to the cunrent status and/or the agent's activity are 
represented on display 27. This enables more informa- 35 
tlon to be displayed at the same time and more mes- 
sage to be given to the driver. The agent and the 
background may represent different information or mes- 
sage. The background may represent where and how 
the vehicle is driving. For example, a tree shown as a 40 
t>ackground may be bent to the wind (to the left). In 
antiier example, where the vehicle is running on an 
uphill or downhill, such a road condition may be repre- 
sented as a background. 

In accordance with yet another embodiment of the 45 
present invention, even when tiie same communication 
. program is executed, the agent's appearance and/or 
activity will be changed depending on the current status 
of tiie vehicle, the driver and tiie passengers, etc. and 
as a result of the study of the cun'ent status and the so 
driver's response, so that the user can make more 
friendly comrnunication with the agertt. 

In this embodiment, processing unit 1 (Rg. 1) is 
connected to agent data memory 29g that stores agent 
program 290, program select table 291 , study item data 55 
292f, response data 293, image data 294, appearance 
select table 308 and familiarity data 300. as shown in 
Rg. 45. Program 290. table 291 and data 292-294 are 



substantially iderrtical to those stored in agent data 
memory 29 of the first embodiment of Rg. 1. Accord- 
ingly, explanation thereof is not repeated herein. The 
familiarity data 300 has also been desaibed in detail In 
refererK^e to Rg. 22. The image data 294 in this enixxi- 
iment stores a great numt>er of appearance aich as 
looks, features, wears, belongings, situation, posture, 
attitude, etc. to be applied to the agent appearing on dis- 
play 27. When the driver selects his or her favorite char- 
acter through input device 22 or any other input means, 
the selected agent appears on display 27 with appear- 
ance selected in reference to the appearance select 
table 308. 

Rg. 46 is an example of the appearance select 
table 308. When the vehicle current status such as 
described in the left-hand columns of the table. 308 is 
detected by a con'esponding sensor of the current sta- 
tus sensor 40 (Rg. 1). a specific appearance of tiie 
agent at the neighborhood center column should be 
selected. The agent's appearance may also be deter- 
mined by the past record study stored in tiie study item 
data 292 and/or the response data 293, in which case 
such data contents should be described in the left-hand 
columns. Each appearance selectable in the table 308 
has the order of priority When a plurality of appear- 
ances are applicable which can not be executed at the 
same time, one of tiiem having the earlier order of prior- 
ity is selected. For example, when the wiper is working 
on Christmas Eve, appearances of "wearing a raincoat", 
"wearing a sweater" and "dressed as Santa Claus" are 
applicable as the agent' wear, in which case the appear- 
ance of "wearing a raincoat" is selected according to pri- 
ority so that tiie agent wearing a raincoat appears on 
display 27. If tiie wiper is turned off. the appearance of 
tiie next priority is selected so that tiie agent appears 
under the disguise of Santa Clause. 

The contents of tiie appearance select table 308 
shown in Fig. 46 Is only example and of course not imi- 
tative. For example, it may be possible that the type of 
tiie vehicle is detected when tiie ignition switch is turned 
on, and the agent wears a formal suit when tiie vehicle 
is a high-grade sedan whereas the agent wears a T-shirt 
when the vehicle is a RV (recreation vehicle) or SUV 
(sport utility vehicle). In anotiier example, tiie agent's 
height changes with the fuel remainder. The agent may 
jump up when VSC or ABS becomes operative. The 
agent may be inclined while excessive acceleration, 
deceleration or lateral gravity is applied to the vehicle. 
The agent may appear with a card indicating television 
or radio channels when the driver turns tiie television or 
radio on. The agent may become tall or fat as the driver 
becomes more familiar to the navigation system. The 
agent's wear or otiier appearance may be changed 
depending on the goal. The agent may perspire when 
tiie room temperature is higher than tiie open air tem- 
perature by a predetermined difference. The agerrt' 
shoes may be changed depending on the road condi- 
tion, more particularly tiie agent wears a pair of moun- 
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taine^ing or trekking boots while the vehicle is running 
on an unpaved road. The environmental data at the left- 
hand column may include the traffic jsun. rainfall, snow- 
fall, sunshine, the number of passengers, etc. The 
user's data may include the driver's occupation. For 5 
example, the agent wears a while overall when the 
driver is a doctor. The user's data may also include the 
driver's or passenger's birthday, payday, blood type, 
annual income, family members, nationality, native lan- 
guage, religion, favorite music, favorite sport, etc. Data 70 
regarding date and time may include anniversary, holi- 
days, lunch time, coffee brake time, dinner time. etc. 
which is known to the driver by a pertinent agent's 
appearance. 

The main routine of the agent operation of this is 
embodiment is the same as shown in Rg. 1 5 and, there- 
fore, description regarding the flowchart of Rg. 15 
should be referred to In understanding the agent control 
operation of this embodiment. In this embodiment, how- 
ever, the agent control operation that is can^ied out at 20 
S114 of the main routine of Fig. 15 is processed as . 
shown by the flowchart of Fig. 47. In the flowchart of 
Fig. 47, steps S601-602 are substantially identical to 
S14-15 of the flowchart of Fig. 9, so that description of . 
the latter steps should be referred to in understanding 2s 
the S601-602 of this flowchart. 

Alter confirming that the applicable communication 
program should be executed in this operation (No at 
S602). agent processing unit 11 discriminates if the 
applicable communication program is an instrument 
control program, at step S603. The instrument control 
program means a program to be executed to control 
operation of the vehicle or any instrument mounted to or 
equipped in the vehicle. If this is the case (Yes at S603), 
the said program is executed to can^y out necessary 
operation under control, at S609. The result of the con- 
trol operation is stored in the study item data 292. 

When the applicable communication program is not 
an instrument control program (No at S603). at step 

5604. agent processing unit determines an agent's 
appearance, In reference to the appearance select table 
308 (Fig. 46), from the cun-ent status (vehicle data, nav- 
igation data, environmental data, user's data, current 
date and time, result of driver's voice recognition, 
driver's familiarity to the agent system etc.). The appli- 
cable communication is executed by displaying the 
agent with the selected appearance, on display 27, at 

5605. The agent may also talk to the driver through 
speaker 25 in accordance with the contents of the appli- 
calbe communication program. Thus, the agent's activ- 
ity (action and voice) such as shown In Figs. 48B. 49B 
and 50 is shown on display 27 with a suitat^e appear- 
ance that is pertinent to the agent's activity or the cur- 
rent status. 

When this communication program is not a ques- 
tioning or responsive one requiring the driver's any 
response (No at S606), the procedure is returned to the 
main routine. When this is the case (Yes at 8606), agent 
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processing unit 1 1 acquires the driver's response or 
answer through microphone 26 and voice processing 
unit 14 or through the driver's operation of input device 
22. at step S607. The acquired response is incorpo- 
rated into response data 293 at step 8608. then the pro- 
cedure is returned to the main routine. In this 
embodiment, especiaily, agent processing unit 11 
acquires at 8607 the driver's response to the agent's 
activity in the executing communication program and a 
time elapse from when the agent' activity is outputted 
until when the driver's response is confinmed. The data 
accumulation or the past record study at 8608 has been 
described in connection with the preceding embodi- 
ments. In this embodiment, when the executed commu- 
nication program is a questioning or responsive 
program that needs the driver's response, the response 
obtained at 608 is incorporated into the response data 
293. When the response data 293 already has full 
records, the oldest record is deleted and replaced by 
the newly obtained record. Further, the familiarity data 
300 (Rg. 22) is renewed to a new value, as in the man- 
ner having been described hereinbefore in connection 
with the preceding embodiments. 

The agent's appearance and activity in this embod- 
iment will be described hereunder in reference to Figs. 
48A-50 by vyay of example. Rgs. 48A-48C illustrates an 
example of the agent control operation executed after 
the ignition switch is turned on. As shown in Fig. 48A, 
agent processing unit 1 1 acquires, as the current status, 
30 a time elapse T1 from the ignition-on from a detection 
signal from a timer (one of sensors 40). It has also been 
confirmed that current vehicle speed Is 35kg/h. the 
driver's voice is recognizable, the wiper is working, the 
current season is summer, the current driver's familiarity 
35 is 27. etc. Further, agent processing unit 1 1 make refer- 
ence to the response data 293 to confirm that the driver 
showed "acceptance" to the agent's activity in the latest 
two application of the program No. 00123. The cun-ent 
default value TO stored in the study item data 292 is 120 
40 minutes. Then, rt has also been confirmed that the 
elapsed time 11 ^ TO. 

From these data obtained and confirmed, agent 
processing unit 1 1 finds the applicable communication 
program No. 00123 (Yes at 8601) and confirms that 
45 application of this program should not be suspended at 
this time (No at S602). It also confirms that this program 
is not an instrument control program (No at 8603). 

Then, agent processing unit 11 determines the 
agent's appearance from the current status of Fig. 48A. 
50 at 8604. The appearance select table 308 of Rg. 46 
teaches that, in these cun-ent status, the appearances 
of "hair falls down naturally", "full face". 'Swear a swim- 
suir. and "tall" should be selected. Accordingly, when 
executing the program No. 00123 at 8605, a tall agent 
55 wearing a swimsuit (or a sportswear) stands naturally 
with her full fact, on display 27. At the same time, she 
talks to the driver "Will you have a rest?" through 
speaker 25. 



23 




EP0 893 



Since this program is a questioning program (Yes at 
606). the driver's response \s obtained at S607, as 
shown in Fig. 48C. The driver's response "neglect", 
"refusal" or "acceptance", together with date and time of 
the respmse. is ^ored in the response data 293 of the 5 
program Na 00123 at S608. The oldest response data 
is replaced with the newly acquired response data. 
When the driver shows "neglect", the default value 
increased by 10 minutes Is applied in the next but one 
application of this program. In case of "refusal", the w 
default value increased by 20 minutes is applied in the 
next but five application of this program. In case of 
"acceptance", the same default value is applied in the 
next application without suspension. These are the 
result of the past record study to be incorporated in the is 
response data 293. Then, the procedure is returned to 
the main routine of Rg. 15. 

Fig. 49A illustrates the different current status, in 
which case the agent's appearance such as shown in 
Fig. 49B is selected in reference to the table 308 of Rg. 20 
46. In this case, the appearance of "hair falls down nat- 
urall/. "face in profile", "wear a sweater", and "shorT 
should be selected. Accordingly, when executing the 
program No. 00123 at S605, a short agent wearing a 
sweater feces in profile, on display 27. and says "Will 2s 
you have a rest?" through speaker 25. Operation at 
other steps will be substantially tiie same as in tiie 
examples of Figs. 49A-49C. 

In the above-described embodiment, a plurality of 
selectable agent's appearance has respective image 30 
data stored in the image data memory section 294. 
However, it may be possible tiiat one or several image 
data is prepared for one communication program and 
changed by image processing unit 13 in accordance 
witii the current status. Change of the agent's appear- 35 
ance may include change or the agent itself. For exam- 
ple, tiie agent is a rabbit when the vehicle is running at 
a high speed whereas the agent is a turtle during low- 
speed drive. When the current status is tiiat the driver's 
voice is recognizable, the agent may appear on display 40 
27 carrying a namelxjard of "PLEASE" or two name- 
boards of "YES" and "NO" (Rg. 50) for urging the driver 
to make a suitable response to the agent's activity in the 
last communication program. 

45 

Claims 

1 . A device mounted in a vehicle, comprising first sen- 
sor means for detecting predetermined current sta- 
tus; first memory means for storing study data so 
renewable in response to entity of tiie latest current 
status detected by sakJ first sensor means; at least 
one personified agent communicatable witii user in 
the vehicle; second memory means for storing a 
plurality of communication programs; communica- ss 
tion program select means for determining one of 
said communication programs to be executed in 
accordance with ttie cun-ent status detected by said 
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f ETSt sensor means and/or the study data cun'entiy 
stored in said first memory means; agent control 
means for controlling activity of said agent in 
accordance with said selected communication pro- 
gram; agent output means for outputting said 
agent's activity to said user in ttie vehicle deter- 
mined by said agent control means. 

2. A device according to daim 1 wherein said first sen- 
sor means detects at least one vehicle condition 
selected from the group consisting of a current 
vehicle position, a current vehicle speed, a current 
shift position, winker operation, wiper operation, a 
coolant temperature, an engine oil temperature, an 
engine oil pressure, a passenger room tempera- 
ture, an open air temperature, ignition switch oper- 
ation, a brake pedal stroke, an accelerator pedal 
stroke, side brake operation, horn operation, belt 
loosening, a fuel remainder, light or lamp operation, 
seat belt operation, door opening, window opening, 
ABS operation, air-conditioner operation, distance 

' to the preceding and succeeding vehicles and 
decrease of air pressure in tire. 

3. A device according to daim 1 or 2 wherein user's 
own data is induded in the current status detected 
by said first sensor means, said user's own data 
being also considered in determining the communi- 
cation program to be executed. 

4. A device according to claim 1. 2 or 3 which further 
comprises second sensor means for detecting 
user's response to the agent's activity outputted by 
said agent output means, and third memory means 
for storing user's response data detected by said 
second sensor means, said user's response data 
being also conskiered in determining tiie communi- 
cation program to be executed. 

5. A device according to claim 4 wherein sakJ user's 
response data is subjected to study so that said 
study data in said first memory means is also 
renewable in response to entry of tiie latest user's 
response data. 

6. A device according to any one of claims 1 to 5 
wherein said agent output means comprises one or 
both of display means and speaker means. 

7. A device according to any one of claims 1 to 6 
wherein said agent output means at least indudes 
speaker means, utterance pattern of the agent out- 
putted by said speaker means being selectively 
determined by said agent control means. 

8. A device according to any one of claims 1 to 7 
wherein said agent output means at least indudes 
display means, appearance pattem of the agent 
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outputted by said display means being selectively 
determined by said agent control means. 

9. A device according to any one of claims 1 to 8 
which further conrprises agent's character select s 
means for detenmining a specific character to be 
given to the agent, said agent control means deter- 
mining the agent^s activity in consideration of the 
agent* s character determined by said agent's char- 
acter select means. io 

10. A device according to any one of claims 1 to 9 
which further comprises fourth memory means for 
storing data indicating user's familiarity to agent 
operation of the device, said agent control means is 
determining the agent's activity in consideration of 
the agent*s familiarity stored in said fourth memory 
means. 

11. A device according to any one of claims 1 to 10 20 
wherein said agent output means allows a plurality 

of different agents to be distinguishably recognized 
one another by the user in the vehicle. 

12. A device according to any one of claims 1 to 11 2s 
which further comprises fifth memory means for 
storing sub-agent program, said sub-agent pro- 
gram being selected in correspondence to a spe- 
cific communication program stored in said second . 
memory means so that a sub-agent of the selected 30 
sub-agent program Is outputted by said agent out- 
put means together with the main agent of the exe- 
cuted communication program. 

13. A device according to claim 12 wherein said sub- 3s 
agent explains the activity of the main agent. 

14. A de/ice according to claim 12 or 13 wherein said 
sub-agent allows short-cut of operation executed by 
the nrtain agent communication program. 40 

15. A device according to daim 12, 13, or 14 wherein 
said sub-agent program becomes operative when 
data indicating user's familiarity to agent operation 

of the device is greater than a predetermined criti- 45 
cal value. 

16. A device according to any one of claims 1 to 15 
wherein said agent output means at least includes 
display means, a background of said display means so 
being selectively determined so that the agent 
appears on said display means together with the 
selected background. 
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FIG. 3 



Program 


Aeent*s Activities 


No. 


Action 


Utterance 


00001 


Obeisance 


"Nice to meet you. Fm Lisa. May I eo with you ?" 


00002 


Obeisance 


"Nice to meet you. I'm Lisa. May I eo with vou T 


00003 


Obeisance 


"Hello, I'm Lisa I will go with you again. Did you sleep well last night ?" 


00004 


Obeisance 


"Hello. Tm Lisa I will go with vou again.** 


00005 


Obeisance 


"What happens with you ?" 


00006 


Obeisance 


"Did you have a good rest ?" 


00007 


Bow 


"Good morning, Fm Lisa May I go with you ?" 


• 


Bow • 


"Good afternoon. Fm Lisa. Mav I eo with vou ?" 


• 


Bow 


"Good evening. Fm Lisa Mav I eo with you 7" 


• 


Bow 


"Good morning. Fm Lisa. May I go with you 7" 




Bow 


"Good afternoon. Fm Lisa. May I go with you 7" 




Bow 


"Good evening. Fm Lisa May I go with you ?" 




Bow 


"Hello. Fm Lisa 1 will go with you again. Did you sleep well last night ?" 




Bow 


"Hello, Fm Lisa. I will go with vou again. Rest yourself " 




Bow 


"Good morning, Fm Lisa. It*s been a long time. Mav I eo with vou ?" 




Bow 


"Good afternoon. Fm Lisa. IFs been a long time. May I go with vou ?" 


• 


Bow 


"Good evening. Fm Lisa. It*s been a lone time. Mav I eo with vou ?" 


• 


Bow 


"What happens with vou ?" 


• 


Bow 


"Did you have a good rest ?" 


00020 


Sleepy Bow 


"Good morning, Fm Lisa. May I go with you ?" 


■ 


SleeoY Bow 


"Good afternoon, Fm Lisa. May I go with you ?" 


• 


Sleepy Bow 


"Good evening, Fm Lisa. May I go with you ?" 


• 


Sleepy Bow 


"Good morning. Fm Lisa. May I go with you ?" 




Sleepy Bow 


"Good afternoon, Fm Lisa. May I go with you ?" 




Sleepy Bow 


"Good evening, Fm Lisa. May I go with vou ?" 




Sleepy Bow 


"Hello. Fm Lisa I will go with you again. Rest yourself " 




Sleepy Bow 


"Good morning. Fm Lisa It's been a long time. May I go with you ?" 




Sleepy Bow 


"Good afternoon. Fm Lisa It's been a long time. May I go with you ?" 




Sleepy Bow 


"Good evening, Fm Lisa It's been a long time. May I go with you ?" 




Cheerfiil 


"Good morning ! Go with me cheerfully. " 




Cheerftil 


"Good afternoon ! Go with me cheerfully. " 




Cheerful 


"Good evening ! Go with me cheerfully. " 




Cheerful 


"Hi f. Go with me cheerfully. " 




Cheerful 


"Hi r Go with me aeain cheerfully. Did vou have a eood rest ?" 




Cheerful 


"Rest yourself. Go with me again cheerfully. " 




Cheerful 


"Good morning ! It's been a long time. Go with me again cheerfullv." 




Cheerful 


"Good afternoon I It's been a long time. Go with me again cheerfuUv." 




Cheerful 


"Good evening 1 It's been a tone time. Go with me aeain cheerfully. " 




Doubtful 


"What happens with you ?" 




Doubtful 


"Are you alright ?" 




Sleepy 


"Good morning 1 Go with me cheerfully. " 




Sleepy 


"Good afternoon 1 Go with me cheerfully. " 




Sleepy 


"Good evenine ! Go with me cheerfully. " 




Sleepy 


"Hi 1. Go with me cheerfully. " 




Sleepy 


"Rest yourself Go with me again cheerfully. " 




Sleepy 


"Good morning 1 It's been a long time. Go with me again cheerfully. " 




Sleepy 


"Good afternoon ! It's been a long time. Go with me again cheerfully. " 




Sleeov 


"Good evenine 1 It's been a lone time. Go with me aeain cheerfully." 
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1 1 1 1 
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1996.01.25 10:05 A : 1997.04.22 18:05 C ! • • • 
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• 
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VOICE PATTERN SELECT TABLE 297 
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ACTION 


UTTERANCE 


02000 


Right, hand is pulled back. 


"the side brake is still in operation. Release it, please.** 


02001 


Right hand is pulled back. 


"Oh, nol The side brake is still working. Release it, 
please." 


02002 


Right hand is pulled back 
with tearful face. 


"Ouchl The side brake is still in operation. Release it, 
please." 


02003 


Riglil hand is pulled back. 


"May I suggest you releasing the side brake." 
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